1 Introduction
Corpora with syntactic annotation (treebanks) are useful resources for training and evaluating statistical parsers, which in turn can be used in different Natural Language Processing (NLP) applications, such as machine translation, information extraction or opinion mining (Gimpel and Smith Reference Gimpel and Smith2014; Nguyen, Moschitti and Riccardi Reference Nguyen, Moschitti and Riccardi2009; Socher et al. Reference Socher, Perelygin, Wu, Chuang, Manning, Ng and Potts2013). Furthermore, studies in corpus linguistics also benefit from the availability of treebanks, which allow researchers to extract information from real linguistic data (McEnery and Hardie Reference McEnery and Hardie2011). However, manually labeling a new corpus is an expensive and time-consuming task, which requires a large effort by expert annotators to obtain high-quality data.
Aimed at alleviating the effort of creating a new treebank, this paper investigates the impact of different factors on cross-lingual parsing (i.e., analyzing a target language with resources from one or more source languages). Thus, we perform several experiments on cross-lingual parsing of Romance languages, and verify the practical usefulness of the lessons learned by carrying out a case study: the creation of a new treebank for Galician.
In the last few years, several strategies for projecting the syntactic annotation from a source language to a target one have been implemented, in order to automatically obtain corpora for the latter language (Hwa et al. Reference Hwa, Resnik, Weinberg, Cabezas and Kolak2005; Ganchev, Gillenwater and Taskar Reference Ganchev, Gillenwater and Taskar2009). The resulting data can then be corrected by an expert, thus reducing the effort with respect to labeling a new resource from scratch. Nevertheless, the different annotation guidelines used in each language resource make it complicated to leverage the cross-lingual resources.
A different approach consists in creating a parser for the target language without the need of a treebank. Some authors apply unsupervised methods (Klein and Manning Reference Klein and Manning2004), while others rely on the direct transfer of model parameters from one language to another (Zeman and Resnik Reference Zeman and Resnik2008; McDonald, Petrov and Hall Reference McDonald, Petrov and Hall2011). These methods usually reduce language-specific information by ignoring the lexical features in the learning process, thus building delexicalized parsers.
More recently, several approaches have emerged with the aim of harmonizing the annotation of syntactic dependencies among different languages and treebanks (McDonald et al. Reference McDonald, Nivre, Quirmbach-Brundage, Goldberg, Das, Ganchev, Hall, Petrov, Zhang, Täckström, Bedini, Bertomeu Castelló and Lee2013; Zeman et al. Reference Zeman, Dušek, Mareček, Popel, Ramasamy, Štěpánek, Žabokrtskỳ and Hajič2014; de Marneffe et al. Reference de Marneffe, Dozat, Silveira, Haverinen, Ginter, Nivre and Manning2014), ending up with the creation of the Universal Dependencies (UD) initiative (Nivre et al. Reference Nivre, de Marneffe, Ginter, Goldberg, Hajič, Manning, McDonald, Petrov, Pyysalo, Silveira, Tsarfaty and Zeman2016). The promoters of this project developed a set of common (universal) guidelines for annotating a treebank, thus facilitating the leverage of syntactic resources as well as the linguistic analysis between languages.
In this respect, some experiments using treebanks of different languages with harmonized labeling showed that cross-lingual parsing can achieve better performance than previous unsupervised approaches (McDonald et al. Reference McDonald, Petrov and Hall2011). However, even though the research on cross-lingual parsing has increased (Agić et al. Reference Agić, Tiedemann, Merkler, Krek, Dobrovoljc and Moze2014; Rosa and Žabokrtskỳ Reference Rosa and Žabokrtskỳ2015b; Tiedemann Reference Tiedemann2015), it is still difficult to answer questions such as the following:
-
• Is lexical distance between the source and target languages more influential for cross-lingual parsing than their structural differences?
-
• Is it worth it to delexicalize the models for cross-lingual parsing of Romance languages?
-
• To what extent can we trust cross-lingual parsing between languages from the same linguistic family?
In order to answer these questions, this paper presents a set of experiments concerning cross-lingual parsing between Romance languages. First, we analyze the lexical distance as well as the mutual lexical coverage between eight linguistic varieties. Then, we evaluate the performance of direct cross-lingual parsers, using both one source language and several treebank combinations.
We will show that, in most cases, delexicalization is not useful for cross-lingual parsing between Romance languages, and this phenomenon is in significant correlation with the lexical coverage of the treebanks of the source and target languages. Furthermore, the results of direct transfers using a single treebank as source achieve a performance equivalent to the use of a given amount of manually annotated tokens of the target language, and this amount can be noticeably increased if a previous selection and adaptation of the best source treebanks is performed.
As previously mentioned, the results of these experiments allowed us to reduce the effort of creating a new UD treebank for Galician. This new resource was manually corrected after an automatic annotation using cross-lingual parsing, confirming that it is possible to train a high performance UD parser for a new language with little manual effort.Footnote 1
The remainder of this article is organized as follows. Section 2 gives an overview of the design process of the UD, and introduces related work on cross-lingual parsing. Then, Section 3 presents a set of parsing experiments using different treebanks of Romance languages. After that, the creation steps of the new treebank for Galician are shown in Section 4, together with several monolingual and cross-lingual tests. Finally, we discuss the main results of this paper in Section 5, and present the conclusions and further work in Section 6.
2 Related work
This section includes a brief introduction to the UD initiative, followed by a presentation of some of the most influential papers concerning cross-lingual parsing.
2.1 Universal dependencies
The UD project—which stems from the Universal Treebanks promoted by Google (McDonald et al. Reference McDonald, Nivre, Quirmbach-Brundage, Goldberg, Das, Ganchev, Hall, Petrov, Zhang, Täckström, Bedini, Bertomeu Castelló and Lee2013)—started in 2014 with the main goals of developing a cross-linguistically consistent grammatical annotation, as well as providing treebanks labeled using the same guidelines.Footnote 2 This harmonized annotation supports multilingual research in both comparative linguistics and NLP. In practice, UD unifies several attempts that had been developed, aimed at performing cross-lingual POS-tagging and dependency parsing.
The representation of UD includes, for each token, its lemma, POS-tag, morphological features and the syntactic dependency it belongs to in the sentence (all this information encoded with universal labels and tokenized using the same criteria). The UD POS-tags were started by Petrov, Das and McDonald (Reference Petrov, Das and McDonald2012), which proposed a tagset of twelve elements, then enriched and modified to the current inventory of seventeen tags.Footnote 3 As these labels only classify POS categories, a different layer encodes the morphological information. In this regard, Zeman (Reference Zeman2008) had presented a tool for converting morphological features from different languages into a universal standard, Interset, later employed in various projects such as HamleDT (Zeman et al. Reference Zeman, Dušek, Mareček, Popel, Ramasamy, Štěpánek, Žabokrtskỳ and Hajič2014).
HamleDT introduced a compilation of twenty nine existing treebanks automatically converted to a harmonized annotation. The syntactic labeling of the first version was inspired by the Prague Dependency Treebank (Bejček et al. Reference Bejček, Panevová, Popelka, Straňák, Ševčíková, Štěpánek and Žabokrtskỳ2012), being adapted to the Stanford dependencies in further versions (Rosa et al. Reference Rosa, Masek, Marecek, Popel, Zeman and Zabokrtskỳ2014).
The Stanford dependencies, initially developed for English (de Marneffe, MacCartney and Manning Reference de Marneffe, MacCartney and Manning2006) and basis of UD, experimented an evolution toward a universal set of dependency relations (de Marneffe and Manning Reference de Marneffe and Manning2008; de Marneffe et al. Reference de Marneffe, Dozat, Silveira, Haverinen, Ginter, Nivre and Manning2014), which facilitated the combination with the mentioned universal POS-tags (e.g., in the already mentioned Google Universal Treebanks) and also with a universal set of morphological features (Tsarfaty (Reference Tsarfaty2013), which proposed a variation of the Standard dependencies, called U-SD) in order to develop the current version of UD.
Therefore, currently, UD merges these different approaches for developing multilingual treebanks with consistent annotation (Nivre et al. Reference Nivre, de Marneffe, Ginter, Goldberg, Hajič, Manning, McDonald, Petrov, Pyysalo, Silveira, Tsarfaty and Zeman2016), using a new version of the CoNLL format called CoNLL-U. It is worth noting that although UD promotes a universal set of syntactic dependencies, it permits the use of other labels for representing language-specific phenomena. These labels, however, are subtypes of the core UD relations (named as udrelation:subtype), so an alignment between universal and language-specific dependencies is preserved in some way.
In sum, the unification of UD lies in the use of (i) universal tagsets for POS-tagging, morphological encoding and dependency syntax, and (ii) common guidelines for tokenizing and for labeling syntactic phenomena. In this regard, UD suggests a unified annotation of controversial structures such as coordination, verbal groups or the relation between a preposition and a noun phrase, among others.
2.2 Cross-lingual parsing
The different strategies that have been used for cross-lingual parsing can be classified in two main groups: (a) data transfer and (b) model transfer approaches. The first one creates artificial data of the target language by projecting the linguistic information from a source treebank, sometimes with the help of machine translation. Differently, model transfer approaches use the source data for training models that can be used for analyzing one or more target languages. As pointed out in Section 1, the emergence of UD facilitates research in cross-lingual dependency parsing, but several papers had already addressed this topic from different perspectives.
2.2.1 Data transfer and annotation projection
Concerning the projection of syntactic labeling, one of the most common strategies is the use of parallel corpora, which was introduced by Yarowsky, Ngai and Wicentowski (Reference Yarowsky, Ngai and Wicentowski2001) for other NLP tasks such as POS-taggers, chunkers, or lemmatizers.
Hwa et al. (Reference Hwa, Resnik, Weinberg, Cabezas and Kolak2005) parse the English version of English–Spanish and English–Chinese parallel corpora, and then project the syntactic dependencies from the source language to Spanish and Chinese, respectively. After that, they train statistical parsers on the resulting data. Even though this strategy requires parallel corpora (which are not easy to obtain for many languages), the best results of the Spanish transfer were better than those obtained by a commercial parser (≈72 percent of unlabeled F-score). However, the Chinese results were noticeably lower (≈44 percent) due to the complexity of the English–Chinese parallel corpora alignment.
The strategy presented by Hwa et al. (Reference Hwa, Resnik, Weinberg, Cabezas and Kolak2005) can be improved in several ways: Smith and Eisner (Reference Smith and Eisner2009) show that using quasi-synchronous features and some manually annotated sentences of the target language provides a boost equivalent to doubling the number of target trees. Another strategy for improving the use of parallel corpora consists in taking advantage also of target trees with partial analysis, since Hwa et al. (Reference Hwa, Resnik, Weinberg, Cabezas and Kolak2005) only used the sentences with perfectly conserved dependencies (Ganchev et al. Reference Ganchev, Gillenwater and Taskar2009). Besides, Ganchev et al. (Reference Ganchev, Gillenwater and Taskar2009) also add some rules for reducing the most frequent differences between some treebanks (e.g., the selection of the main and auxiliary verb in verb groups or the status of the prepositions in noun phrases).
More recently, various approaches addressed again annotation transfer between parallel corpora, taking advantage of the emergence of resources with harmonized labeling. Thus, Tiedemann (Reference Tiedemann2014) projects dependency labels using both manual translations (from Europarl) and machine translated corpora, showing that a consistent annotation between treebanks improves the performance of the transfer. Similar experiments, including data subset selection, are presented by Tiedemann (Reference Tiedemann2015a), which confirm that building parallel corpora with machine translation gives better results than projecting the labels in automatically annotated parallel corpora. The use of machine translation to obtain labeled data of a target language was also addressed in several works, showing the importance of the lexical features and the impact of POS-tagging in dependency parsing (Tiedemann, Agić and Nivre Reference Tiedemann, Agić and Nivre2014; Tiedemann and Agić Reference Tiedemann and Agić2016).
The use of dense projected structures is presented by Rasooli and Collins (Reference Rasooli and Collins2015), to obtaining high-quality projections that improve cross-lingual parsing performance.
Following a similar approach to the one presented by Agić, Hovy and Søgaard (Reference Agić, Hovy and Søgaard2015) for POS-tagging, Agić et al. (Reference Agić, Johannsen, Plank, Martínez Alonso, Schluter and Søgaard2016) perform cross-lingual parsing for languages with very low resources. Both for POS-tagging and parsing, the authors rely on multi-source strategies for projecting the labeling of widely translated texts, such as the Bible.
Finally, Lacroix et al. (Reference Lacroix, Aufrant, Wisniewski and Yvon2016a) carry out annotation transfer using parallel corpora, ignoring unattached words and many-to-many alignments between the two resources. This paper shows that learning from high-quality (but partial) data is better than utilizing fully annotated data with some noise. Using the same approach, Lacroix, Wisniewski and Yvon (Reference Lacroix, Wisniewski and Yvon2016b) analyze the impact of pre-processing (and post-processing) the parallel data, proving that filtering out noisy sentences improves cross-lingual parsing. Also, they address multi-source transfer of dependency annotation, achieving better results when combining treebanks from the same linguistic family, and they show that the transfer results are surpassed by supervised models trained on ≈300 sentences (depending on the languages).
2.2.2 Model transfer
As mentioned, the other main strategy for parsing a new language consists in using cross-lingual models built with resources from other linguistic varieties.
The adaptation of a parser aimed at analyzing a similar language is addressed by Zeman and Resnik (Reference Zeman and Resnik2008), who evaluate the use of Danish corpora to train a parser for analyzing Swedish. The best results are obtained when performing a delexicalization of the corpora (replacing the words with their POS-tags, previously mapped between the two languages), concluding that this strategy produces the same results as manually annotating 1,546 sentences in the target language.
McDonald et al. (Reference McDonald, Petrov and Hall2011) were one of the first researchers using universal POS-tags for dependency parsing, also introducing the multi-source approach for cross-lingual parsing. They train delexicalized models that obtain better results than unsupervised approaches, and show that multi-source parsers (built with simple concatenation of the training corpora of different languages) can be useful for cross-lingual parsing. A similar approach was presented by Cohen, Das and Smith (Reference Cohen, Das and Smith2011), who combine supervised models of various source languages for both POS-tagging and dependency parsing.
Søgaard (Reference Søgaard2011) trains delexicalized parsers selecting—in the source treebanks—only those sentences whose structure is similar to the target language, obtaining better performance than the previous method, also for non-related languages (Bulgarian, Portuguese, Arabic, and Danish).
Naseem, Barzilay and Globerson (Reference Naseem, Barzilay and Globerson2012) implement an algorithm for transferring dependency models that learns different properties from multilingual treebanks, even from non-related languages. The system first learns the universal distribution of each POS-tag’s dependents, followed by an ordering component that determines the position (left or right) of each dependent. The results on several languages largely outperform direct delexicalized parsers as well as the concatenation of multiple source treebanks.
Täckström, McDonald and Uszkoreit (Reference Täckström, McDonald and Uszkoreit2012) and Durrett, Pauls and Klein (Reference Durrett, Pauls and Klein2012) also rely on delexicalized parsers. The former performs an enrichment of the syntactic transfer through cross-lingual word clusters used as features, while the latter adds lexical features by means of bilingual dictionaries, increasing the accuracy of the cross-lingual parsing between 1 percent and 2 percent. A similar approach performs relexicalization on multi-source parsers built by means of selective parameter sharing (Täckström, McDonald and Nivre Reference Täckström, McDonald and Nivre2013).
Several cross-lingual parsing experiments of related languages (Croatian, Serbian and Slovene) were performed by Agić et al. (Reference Agić, Tiedemann, Merkler, Krek, Dobrovoljc and Moze2014), suggesting that delexicalization is not necessary for cross-lingual parsing in these Slavic languages.
McDonald el al. (Reference McDonald, Nivre, Quirmbach-Brundage, Goldberg, Das, Ganchev, Hall, Petrov, Zhang, Täckström, Bedini, Bertomeu Castelló and Lee2013) presented the Google Universal Dependency Treebanks, the first widely adopted set of harmonized treebanks, providing a more reliable evaluation of cross-lingual parsing. The experiments performed in that paper show that for each of the Germanic and Romance languages analyzed (German, English, Swedish, Spanish and French), the best source is from the same linguistic family.
Using the first version of the UD treebanks, Tiedemann (Reference Tiedemann2015b) performs an exhaustive evaluation of cross-lingual parsing, measuring the impact of predicted POS-tags (when compared to gold ones), and also carrying out some experiments in annotation projection and treebank translation.
With the HamleDT treebanks, Rosa and Žabokrtskỳ present a metric for measuring the distance between languages (Rosa and Žabokrtskỳ Reference Rosa and Žabokrtskỳ2015). This distance is then used to assign a weight to each of the source treebanks in a multi-source scenario, optimizing the training data to the target language (Rosa and Žabokrtskỳ Reference Rosa and Žabokrtskỳ2015b).
In a similar way to Naseem et al. (Reference Naseem, Barzilay and Globerson2012) or Täckström et al. (Reference Täckström, McDonald and Nivre2013), Zhang and Barzilay (Reference Zhang and Barzilay2015) address multilingual transfer parsing, taking advantage of hierarchical tensor models. In order to incorporate (partial) lexical information, they use multilingual word-embeddings of the most frequent words. Following the distributional semantic approach, other works learn bilingual word-embeddings from parallel corpora to avoid the problems of delexicalization in multi-source cross-lingual parsing (Guo et al. Reference Guo, Che, Yarowsky, Wang and Liu2015; Guo et al. Reference Guo, Che, Yarowsky, Wang and Liu2016).
Duong et al. (Reference Duong, Cohn, Bird and Cook2015) present a neural network approach for cross-lingual parsing of low-resource languages. The method uses an interlingual representation with some specific mappings for each language, and it also infers syntactic information from multilingual word-embeddings. Even if it is not mainly focused on syntactic analysis, Søgaard et al. (Reference Søgaard, Agić, Martínez Alonso, Plank, Bohnet and Johannsen2015) showed how bilingual word-embeddings learned from Wikipedia (without using parallel corpora) can be useful for cross-lingual parsing.
Aufrant, Wisniewski and Yvon (Reference Aufrant, Wisniewski and Yvon2016) use linguistic information from the World Atlas of Language StructuresFootnote 4 to adapt the sentences of delexicalized source treebanks to the structure of a target language (e.g., word order, use of determiners, etc.). This strategy obtains better results than using a POS-tag model of the target language.
Recent experiments also combined several source treebanks for training multilingual parsers, capable of analyzing texts in more than one language (Vilares et al. Reference Vilares, Alonso and Gómez-Rodríguez2016). These approaches can be implemented without performing delexicalization of the training data, so the resulting parsers effectively use lexical information from one language to analyze a different one.
In the present paper, we focus on the analysis of different syntactic properties and lexical similarity of the source and the target languages for cross-lingual parsing of Romance languages. Some of the results of this analysis are then applied in a case study, the construction of a new UD treebank of Galician.
3 Cross-lingual transfer of parsing models for Romance languages
As pointed out in the previous section, the emergence of harmonized treebanks for several languages has allowed the research community to evaluate the transfer of syntactic resources between different linguistic varieties. However, several experiments have shown that cross-lingual parsing results are not always satisfactory.
In this respect, the experiments by McDonald et al. (Reference McDonald, Nivre, Quirmbach-Brundage, Goldberg, Das, Ganchev, Hall, Petrov, Zhang, Täckström, Bedini, Bertomeu Castelló and Lee2013) suggest that Romance languages might be reasonably well analyzed using resources from other varieties from the same linguistic family (e.g., a parser for Spanish obtained >75 percent labeled attachment score (LAS) analyzing French data). As the cross-lingual results on Germanic languages are lower than those of the Romance ones, the authors’ hypothesis is that much of the divergence in Romance languages is lexical (and not structural).
Taking the above into account, this section includes a detailed exploration of cross-lingual UD parsing in Romance languages, analyzing the impact not only of lexical differences but also of other divergences such as the amount of training data or the number of dependency labels utilized by the annotators. To perform the experiments we used the 1.3 version of the UD treebanks for Romance languages: Catalan (CA), Castilian Spanish (ES), French (FR), Italian (IT), Romanian (RO), European Portuguese (EP) and Brazilian Portuguese (BP).Footnote 5
It is worth noting that, as UD is an ongoing project, some of the available treebanks present divergences in annotation, since they derive from previous corpora labeled following diverse guidelines. In this regard, differences in tokenization (e.g., the current version of the BP corpus do not split contractions, as proposed by UD) or in the use of some dependencies (e.g., ES and EP treebanks do not use the expl dependency) could have an effect in cross-lingual parsing tests.Footnote 6
Concerning the experiments, we first show the results of lexical similarity and coverage tests aimed at estimating the lexical distance between the analyzed languages. Then, we carry out a set of cross-lingual parsing evaluations between all the mentioned Romance languages, using both lexicalized and delexicalized models. Finally, we calculate the learning curve for each language, and verify the amount of training data in the target language needed for outperforming cross-lingual parsing.
3.1 Lexical similarity and coverage between Romance languages
The lexical distance between two Romance languages (the source and the target) may be important in their mutual cross-lingual parsing. To find out the impact of this distance, we calculated both the lexical similarity and the treebank coverage of every language pair. The first experiment gives us an approximation of the general lexical distance between two languages, while the coverage analysis puts the focus on the frequency of co-occurrence of the words in the source and target corpora.
We used two different strategies for computing the lexical similarity between two languages and to obtain the lexical coverage of their treebanks. For the first analysis, we exploited large dictionaries of each language (namely those provided by the latest version (4.0) of FreeLing (Padró and Stanilovsky Reference Padró and Stanilovsky2012),Footnote 7 together with the DELAF_PB—for Brazilian Portuguese (Muniz, Nunes and Laporte Reference Muniz, Nunes and Laporte2005) and the MULTEXT—for Romanian (Erjavec Reference Erjavec2012)), to obtain a general comparison between the language pairs. The results of this analysis are shown in Table 1, where each row contains the percentage of tokens of the target dictionary (in each column) that are covered by the source one. For instance, the European Portuguese dictionary covers 20.6 percent of the Spanish one (i.e., 20.6 percent of the Spanish words appear in the EP lexicon). It is worth noting that these values refer to the matching of orthographic tokens (and not of lexical entries from a linguistic point of view), since tokens are used as features by probabilistic parsers.Footnote 8
Lexical similarity between Romance languages computed using large dictionaries. Numbers in italic are monolingual values (same source and target dictionaries)
Each row shows the coverage percentage of a source dictionary on the target ones (in the columns). The last column is the number of entries of each dictionary.
For computing the lexical coverage between the treebanks, we took advantage of the train splits of the 1.3 UD treebanks, also used for training the cross-lingual parsers.Footnote 9 The numbers in Table 2 are the percentages of word occurrences of one treebank (in each column) that are covered by the source treebank (in each row).Footnote 10 For instance, this analysis concludes that 55.4 percent of the Spanish word occurrences (in the treebank) can be covered by a model trained with the Catalan data. Therefore, these results are more related to cross-lingual parsing than the previous experiment, which analyzes lexical distance from a more generic point of view.
Lexical coverage (case sensitive) between Romance languages computed using the train splits of the UD treebanks (version 1.3). Numbers in italic are monolingual values (same source and target dictionaries)
Each row shows the coverage percentage of a source treebank on the target ones (in the columns). The last columns show the average coverage (Avg), and the number of unique token–TAG elements (Words) of each treebank, respectively.
The results of Table 1 show that Spanish, both varieties of Portuguese and Galician are the languages with a closest relation in terms of lexical units, followed by Catalan. This is not strange due to their geographical closeness (all of them are Iberian Romance languages), even if they use different spelling traditions. In contrast, the dictionaries of French (and also of Italian and Romanian) have a very low coverage (<2 percent) of the other Romance languages.
However, these large differences are reduced if we take into account the word frequency used by the treebank coverage analysis (Table 2), which also minimizes the impact of variations in the size of the resources.Footnote 11 In absolute terms, several Romance languages have a few coincident words (token and POS-tag) between them, but some frequent function words (and also nouns and adjectives) co-occur in different languages, a fact that, in principle, will favor the model transfer between them. In this respect, Spanish seems to be the language with higher mutual coverage among the Romance languages (bearing in mind that there are two varieties of Portuguese in the analysis).Footnote 12 These numbers also show other common linguistic perceptions, such as the relatedness between CA and FR, GL and EP, or IT and ES. Finally, the coverage values of Romanian are lower (and more homogeneous) than those of the other varieties: the highest coverage value as target is of 25.5 percent (CA), and as source language it reaches 34.9 percent (EP). While the dictionary-based comparison classifies French as the least related language (followed by Italian and Romanian, at the same level), Romanian is the language with least lexical coverage (it covers on average ≈24 percent of the other languages, versus 31 percent and 33 percent of Italian and French, respectively) when comparing the treebank lexica.
In order to estimate the impact of the similarity and mutual coverage values, we will come back to these numbers when evaluating the cross-lingual transfer results.
3.2 Experiments on cross-lingual parsing
In the following, we show the results of several experiments concerning cross-lingual UD parsing of Romance languages. We used as training and testing data the splits provided by the UD 1.3 treebanks. For building the models, we utilized MaltParser 1.9 (Nivre et al. Reference Nivre, Hall, Nilsson, Chanev, Eryigit, Kübler, Marinov and Marsi2007) executed out-of-the-box with Nivre’s arc-eager algorithm (Nivre Reference Nivre2004).Footnote 13 The CoNLL-X eval.pl script (version 1.9) was used for computing both the LAS and the unlabeled attachment scores (UAS) (ignoring punctuation tokens for computing the results).
In order to obtain realistic results (Tiedemann Reference Tiedemann2015b), we used predicted POS-tags in the test sets, obtained by UDPipe POS-tagger models previously trained on the train splits of the UD treebanks. Also, the language-specific relations were converted to their universal label in both training and test sets (e.g., acl:relcl → acl).
3.2.1 Direct model transfer
The first group of parsing experiments takes advantage of the UD harmonized treebanks for performing direct cross-lingual syntactic analysis. Thus, we trained both lexicalized and delexicalized models (replacing the tokens with ‘X’) for each Romance language, and applied them directly to each of the other linguistic varieties. We also built several All models, trained with the combinations of all the source treebanks except the target one.
Tables 3 and 4 contain the MaltParser results of these evaluations. The numbers in Table 3 are LAS values, while the UAS results are shown in Table 4. In both tables, the testing treebank is represented in each column, while rows correspond to the training data. Furthermore, each language row includes the lexicalized (top) and delexicalized (bottom) variants. The rightmost columns of Tables 3 and 4 include the size and the number of dependency relations of each training corpus, respectively. The second row of Table 3 shows the precision of the predicted POS-tags.
Cross-lingual parsing results (LAS values) of Romance languages in UD 1.3
Rows correspond to source languages (train), and columns to target languages (test). Each language row contains the results of a lexicalized model (top) and a delexicalized variant (bottom). All models are concatenations of all the training corpora except the target one. Values in bold highlight the best source and the best monolingual source per language, underlined results are those with better performance when delexicalized, and numbers in italic are the monolingual results. The second row includes the precision of the predicted POS-tags in the target languages. The EP column has an additional value in bold (IT) since the best cross-lingual results were obtained with a variety of the same language, BP. The last column shows the size (in number of tokens) of the treebanks.
Cross-lingual parsing results (UAS values) of Romance languages in UD 1.3
Rows correspond to source languages (train), and columns to target languages (test). Each language row contains the results of a lexicalized model (top) and a delexicalized variant (bottom). All models are concatenations of all the training corpora except the target one. Values in bold highlight the best source and the best monolingual source per language, underlined results are those with better performance when delexicalized, and numbers in italic are the monolingual results. The EP column has an additional value in bold (ES) since the best cross-lingual results were obtained with a variety of the same language, BP. The last column shows the number of dependency relation used in each treebank.
The results of the lexicalized models in Table 3 indicate that, when parsing a new language, there are no huge differences in LAS depending on the language used as source. The largest divergence appears in the values on the EP treebank, where the Italian model achieves 65.56 percent while RO obtains 60.90 percent. However, if we momentarily ignore Romanian—whose results are very different to those obtained with the other languages—, the largest difference does not reach 4.5 percent (between CA and ES analyzing FR).
On average, any Romance language might parse a different one with LAS results between 12 percent and 16 percent lower than using the proper treebanks as training data. As mentioned, Romanian is an exception of this behaviour, since neither of the other languages achieves 50 percent LAS on its test data. We have to keep in mind that the Romanian treebanks showed more lexical differences with the resources of the other languages (Table 2), so this fact may have influenced the results. Also, note that the Romanian treebank has, together with FR, the highest number of dependency relations (38).
The UAS cross-lingual results follow similar tendencies to the LAS values, even though the difference between the monolingual and cross-lingual values decrease to an average of ≈9 percent. Furthermore, the divergences among different source languages (parsing the same target) are even smaller when computed using UAS (≈2 percent). Again, Romanian has the worst results, both as source and as target language (except in one case: it works slightly better than Catalan and Brazilian Portuguese analyzing French).Footnote 14 Apart from the large tagset of Romanian (which may have an influence on low LAS results in cross-lingual parsing), other possible factors causing poor UAS results, when compared to the other Romance languages, are structural divergences as well as the importance of lexical features. Related to this, the highest difference between the lexicalized and delexicalized models (both monolingual) occurs again in Romanian (>8 percent LAS and >6 percent UAS).
If we look at the size of the training corpora for cross-lingual parsing, the results suggest that, once we have a certain amount of data (which is the case of the analyzed languages), the quantity might not be critical.Footnote 15 Even though Romanian (which has the smallest training dataset) had the worst results among Romance languages, the models of Catalan (with the largest corpus) do not surpass the results of Spanish, and only had better values than European Portuguese—whose training corpus has half the size—in one case (interestingly, parsing BP).
Concerning the performance of cross-lingual transfer of delexicalized models, Table 5 shows the differences between full and delexicalized variants for each language. In each language row, the top values are LAS, while UAS values are at the bottom. If we analyze the LAS results, only in fifteen out of forty-eight cases of delexicalization produced better results (in UAS, this value increases to twenty). Most of these improvements, again, occur in Romanian, whose delexicalized model works better in every Romance language (except in the LAS results of BP and FR).
Differences between lexicalized and delexicalized models for each language
Rows correspond to source languages (train), and columns to target languages (test). Each language row contains the LAS (top) and the UAS (bottom) results. Underlined numbers denote models with better results when delexicalized. The last column shows the average impact of delexicalization in the cross-lingual experiments.
Furthermore, the drops of the delexicalized models are noticeably higher than the benefits produced by the referred fifteen/twenty parsers. The last column of Table 5 includes the average impact of the delexicalization process (ignoring the monolingual results). On average, all the delexicalized parsers behave worse than the full models in a cross-lingual setting, except for those trained on Romanian, which show average improvements of 2.61 percent (LAS) and 5.85 percent (UAS). Consequently, the results suggest that, at least for Romance languages, removing word features does not seem the best strategy for cross-lingual parsing.
This latter remark, together with the relatively lower coverage values of Romanian, took us to explore a possible correlation among lexical coverage between two languages and the impact of delexicalization. So, we applied Kendall’s tau coefficient (Kendall Reference Kendall1938) in each cross-lingual model, concluding that the coverage values (Table 2) between a source and a target language are significantly correlated with the impact of delexicalization (with p = 0.0023 in LAS, and p = 0.0006 in UAS): the higher the coverage, the lower the benefits of delexicalization, and vice-versa. In this regard, and even if this is not a rigid rule, the results indicate that some of the language pairs with less than ≈35 percent of lexical coverage (European Portuguese–Italian, Italian–Spanish, or different pairs including Romanian) are better analyzed when delexicalized.
Note, however, that although lexical coverage has an impact on delexicalization, it is just one among other factors that influence cross-lingual parsing. As an example, Romanian has better coverage values of European Portuguese than Italian or Catalan, but these latter models produce better results on EP than the RO parser.
3.2.2 Monolingual learning curve versus cross-lingual model transfer
The previous evaluations showed that direct cross-lingual parsing obtains reasonably good results between Romance languages. Therefore, the following set of tests is aimed at knowing to what extent it is needed to manually annotate a treebank for a new language in order to train a statistical parser. Obviously, high-quality manually annotated data for the target language achieves better results than direct cross-lingual transfer, but the labeling process may be very expensive. Taking the above into account, we obtained LAS and UAS learning curves for each language and compared them with the previously explained cross-lingual results.
In order to create the learning curves we built different monolingual models by starting with just 1,000 tokens of training data, and then adding 1,000 more in nine iterations, until achieving 10,000 tokens. After that, we incrementally enlarged the amount of new data, with additions of about 2,000, 5,000, 25,000 and 50,000 tokens, before using the whole training set.Footnote 16
Figure 1 represents the LAS and UAS learning curves of European Portuguese, together with the performance of the full Spanish model parsing the same EP data (horizontal lines). Similarly, the learning curves of Spanish can be seen in Figure 2, which also contains the results of the Italian parser on the Spanish gold standard.
LAS and UAS monolingual learning curves of European Portuguese (0 − 64, 000 tokens) versus LAS and UAS results of the Spanish model in the same EP test data.
LAS and UAS monolingual learning curves of Spanish (0 − 64, 000 tokens) versus LAS and UAS results of the Italian model in the same ES test data.
As can be seen in these two example figures (and in all the other curves that have been created, omitted here due to space reasons), the best single models obtain similar LAS results in cross-lingual parsing to >3, 000 (EP) and >2, 000 (ES) manually annotated tokens of the target language. Interestingly, this value is very similar in all the analyzed languages (once again, except for Romanian, which outperforms both LAS and UAS cross-lingual parsing with just 1,000 tokens).
Concerning UAS, the direct transfer of single models varies depending on the language, achieving performance values comparable to 4, 000–8, 000 tokens of the target language.
Therefore, this evaluation indicates that one could parse, without labeled data of the target language, a new linguistic variety with similar results than those obtained with ≈3, 000 or ≈6, 000 manually annotated tokens, depending on the objectives of the syntactic analysis (LAS or UAS).
It is worth noting that the cross-lingual results shown in the previous learning curves corresponded to single models. In several cases, they could be replaced by the combined (All) parsers, increasing the threshold of needed training data up to ≈9, 000 (LAS) and >16, 000 (UAS) tokens (e.g., in European Portuguese). These values heavily depend on the language, since the combined models do not always perform better than the best single ones (see Tables 3 and 4).
This fact leads us to carry out a previous analysis of the source and the target treebanks in order to build a focused combination that could surpass both single models and the all versus one concatenation. These experiments, which are partially based on the results that have been presented, are introduced in the next section, aimed at reducing the effort of creating a UD treebank for a new language.
4 Galician UD treebank
As has been shown in the previous experiments, Romance languages can be reasonably well parsed using cross-lingual resources built with harmonized annotation, such as universal dependencies. This fact allows researchers to obtain labeled data for a new language without manual annotation. Even if this process does not always provide high-quality resources for training and testing statistical parsers (and also the annotation may be biased (Berzak et al. Reference Berzak, Huang, Barbu, Korhonen and Katz2016), it can be seen as a good starting point for creating a treebank for a new language.
In this regard, this section presents a case study of the development of a new UD treebank for Galician. The treebank was built by means of cross-lingual parsing and manual correction, applying an iterative bootstrapping strategy.
First, we briefly present the main properties of the Galician language as well as of the corpus we used as source, followed by the core guidelines for annotating Galician with UD. After that, we use the Galician data to carry out similar experiments to those performed in Section 3, thus extending the cross-lingual parsing evaluations, verifying whether the new treebank has a similar behavior to that of the other Romance corpora, and testing the proposed approach in a real scenario.
4.1 Galician language
Galician is an Indo-European linguistic variety and a part of the Western Ibero-Romance group evolved from Vulgar Latin. It derives from the medieval Galician-Portuguese language (Teyssier Reference Teyssier1982).
Modern Galician is spoken in the Spanish Autonomous Region of Galicia by about 2.5 million people (Xunta de Galicia 2004), and it is the official language together with Spanish, which has a strong influence on different Galician characteristics such as its syntax, morphology or phonology (Freixeiro Mato Reference Freixeiro Mato2000; Figueroa Reference Figueroa1997).
With an eye on cross-lingual parsing, it is important to note that the current spelling of Galician is based on the Spanish one, and also that some linguists still consider Galician as a variety of (Galician-)Portuguese language (Cintra and Cunha Reference Cintra and Cunha1984), due to its common origin and present similarity. In this regard, some studies have used Portuguese resources for NLP in Galician, taking advantage of the high relatedness between these linguistic varieties (Malvar et al. Reference Malvar, Pichel, Senra, Gamallo and Garcia2010).
So, theoretically, both Portuguese and Spanish treebanks may be the best sources for cross-lingual parsing in Galician, as also the numbers in Tables 1 and 2 reinforce.
4.2 Source corpus
In order to build a UD treebank for Galician, we first selected a corpus with some manually corrected linguistic information as the starting point. This choice allowed us to reduce the effort of manually annotating all the information of the corpus.
The selected resource was XIADA, a Galician corpus with annotation of lemmas as well as POS-tags with rich morphological information reviewed by experts (Rojo et al. Reference Rojo, Martínez, Noya and Barcala2015). The current version of this corpus has 741,833 tokens, and it is divided in four sets, each one belonging to a different typology: (i) generic press, (ii) economic press, (iii) short stories and (iv) some free unrelated sentences. The first subcorpus (of generic press, called xeral) is the first one that we have begun enriching with syntactic information.
We have programmed a script to convert XIADA to the CoNLL-U format, extracting both the UD POS-tags and the morphological features from the original POS tagset.
4.3 Annotation guidelines
Before starting the addition of syntactic information to XIADA, we defined the annotation guidelines for labeling the Galician corpus using the UD 1.4 version. These guidelines are based on three main foundations (Garcia Reference Garcia2016):
-
(1) Use of the UD recommendations whenever possible.
-
(2) Use the smallest possible number of language-specific relations.
-
(3) For labeling structures with more than one possibility of analysis, make the corpus coherent with the European Portuguese and Spanish ones (in this order).
Taking the above into account, the main properties of the UD labeling for the Galician-TreeGal treebank are the following:
-
• Tokenization: The current version of the corpus maintains the original tokenization of XIADA, which joins compound proper nouns and some multiword expressions into single tokens (e.g, ‘John_Lennon’, or ‘a_as_veces’, sometimes). As UD recommends to split these cases, these disagreements are being corrected for adapting our treebank to the UD 2.0 version.
-
• Pseudo-copulative verbs: Verbs belonging to this class are tagged as cop (copulative) when they function as copulae (e.g., ‘Miguel_Barros permanecerá relegado’, Miguel Barros will remain relegated).
-
• Modal, temporal and aspectual verbs: These verbs are considered aux (auxiliary) of the main verb they depend on (e.g., ‘debe conducirnos’, should drive us, or ‘deixa de ser’, stop being). Similarly, auxiliary verbs in verbal periphrasis are also tagged as aux (e.g., ‘vai gañar’, will win).
-
• Years are marked as nmod (nominal modifier). In further versions, they could be labeled as nmod:tmod, a subtype relation used in other UD treebanks for identifying temporal expressions.
-
• Objects: UD 1.4 recommends labeling as dobj (direct object) dative objects when they occur in a sentence with no explicit direct object (‘a tarefansubj corresponderoot lleexpl a o goberno dobj’, the task falls to the government ). Nevertheless, we preferred to mark them as iobj (indirect object) because it facilitates both the linguistic analysis (e.g., transitivity) and the information extraction from the treebank (and from other corpora with automatic parsing), as in the XIADA corpus, both direct and indirect objects can be introduced by the same preposition (usually a): ‘apuntandoroot a o pobodobj’Footnote 17 (pointing to the people), and ‘correspóndelle a o gobernoiobj’ (labeled as dobj above). Note, however, that these cases can be automatically converted to dobj in case it could be needed. Apart from that, we followed the UD recommendation of annotating reflexive, reciprocal and expletive pronouns as expl (expletive).
These guidelines (as well as the UD 1.4 recommendations) were used for labeling the first 1,000 sentences of XIADA.
4.4 Annotation process
Instead of starting the labeling process of XIADA from scratch, we applied a cross-lingual parser trained with a combination of the European Portuguese and Spanish treebanks, due to their similarities with the target language. Some labels of this combined corpus were automatically adapted to the Galician guidelines (we simplified the subtype relations by their core dependency, and automatically replaced the annotation of the reflexive pronouns with expl) in order to avoid the use of unwanted dependencies.
This combination was used as training data for building a cross-lingual parser, then applied to the first ≈1, 000 tokens of the Galician corpus (from the xeral subcorpus). These sentences were manually corrected by one of the authors, and then added to the training corpus for automatically labeling the next ≈1, 000 tokens. This bootstrapping process was repeated until achieving 1,000 sentences (>24, 000 tokens).
4.5 Experiments
Once a gold standard treebank for Galician had been obtained, we performed various tests with the following objectives:
-
• To know what is the best single model for cross-lingual parsing in this language.
-
• To verify to what extent we can adapt and combine source treebanks in order to increase parsing performance.
-
• To analyze the impact of the amount of training data in Galician.
-
• To check how a small treebank behaves in cross-lingual parsing on the same language family.
Note that these evaluations complement those performed in Section 3, providing new information about the cross-lingual parsing in Romance languages.
As in the previous experiments, the parsers were built with the train sets of UD 1.3, using both MaltParser and UDPipe (whose results are not reported here due to their similarity). In the case of Galician, we evaluated the parsing performance using both predicted and gold POS-tags (since the selected corpus already had manually reviewed POS annotation). We used the Galician POS-tagger provided by LinguaKit (Garcia and Gamallo Reference Garcia and Gamallo2015), achieving a precision of ≈93 percent.Footnote 18 For testing, we used all the manually reviewed data as gold standard for Galician, except in a learning curve analysis (Figure 3).
Table 6 shows the cross-lingual results on the Galician data of direct transfer from the other Romance languages. On average, the best results are achieved using the European Portuguese treebank as source, although Italian (especially in LAS) and Spanish also reach high values. The All combination produces better results (both LAS and UAS) than the best single model. In some way the best single results (EP and ES) are in accordance with the linguistic relatedness these languages have with Galician, and also with the lexical distance between them. But these factors cannot be generalized, since, e.g., Italian (whose lexical similarity is lower than most of the other languages) also achieves good cross-lingual results in our gold standard (better than Catalan, with higher lexical coverage).
Cross-lingual results of different parsers of Romance languages on the Galician gold standard, both using predicted and gold POS-tags
In each metric row (pred and gold LAS and UAS), top numbers correspond to full (lexicalized) models, and the bottom ones to delexicalized models. Bold numbers mark the best source languages, while underlined values denote those models with better results in the delexicalized scenario.
The following evaluation analyzes the impact of two adaptations of the source treebanks aimed at approximating them to the target language. First, we applied a tool for converting the Portuguese orthography to the Galician one—which, in turn, is based on the Spanish spelling—, obtaining a new EP treebank with Galician-like orthography (Malvar et al. Reference Malvar, Pichel, Senra, Gamallo and Garcia2010)Footnote 19 : the lexicon of this new treebank covers 82.6 percent of the Galician one, up from 63.4 percent of the original version. The tool was also applied to the European Portuguese dictionary, increasing the similarity with respect to the Galician one from 29.3 percent to 57.5 percent. This new model, built by means of a lexico-orthographic adaptation, is called AP (Adapted Portuguese).
The other adaptation concerns the annotation of the reflexive pronouns in both European Portuguese and Spanish treebanks, which differs from the UD guidelines applied to Galician. In this respect, we replaced the dependency label of these pronouns (dobj or iobj) with expl. These new models are referred as ex variants.
The first columns of Table 7 show that the lexico-orthographic adaption of the EP treebank allows the cross-lingual parsing to increase its performance between 1.17 percent (UAS) and almost 2 percent (LAS), depending on the POS.Footnote 20 These results suggest again that lexical distance has an impact on cross-lingual parsing, even if it is not a decisive factor for transferring a parser from one language to another.
Results of several combined and adapted cross-lingual models on the Galician gold standard, both using predicted and gold POS-tags. Bold numbers highlight the best source models
AP is Adapted Portuguese, and ex means labeling adaptation of the expl relation.
Adapting the labeling of the reflexive pronouns also boosted the European Portuguese and Spanish models, namely the LAS results, a fact that is understandable because it was a simple label change. In this regard, the Spanish and Portuguese ex variants increased their LAS scores by about 0.9 percent and 0.1 percent, respectively.
Concerning the combined models, it is worth noting that the All parser (Table 6) still performs better than these ES and EP adaptations. However, the combination of the ex variants of Spanish and Adapted Portuguese outperforms the All model results by ≈1.1 percent.
Finally, the last evaluated combination adds the Italian treebank—which uses similar guidelines for labeling the expl pronouns as the Galician one—to the AP_ES es data. This new model (named AP_ES_IT ex ) increases both LAS (≈0.4 percent) and UAS (≈0.1 percent) in Galician, achieving the best results: 76.54 percent LAS and 82.43 percent UAS using the gold POS-tags provided by the original corpus.
In order to estimate the impact of the amount of training data in Galician, we created learning curves using a random split of the corrected treebank (with ≈5, 000 lines: 4,922 tokens and 191 sentences) as gold standard, and the remaining (≈20, 000 lines: 19,297 tokens and 809 sentences) for training, adding ≈1, 000 tokens in each iteration.
Figures 3 and 4 show the LAS and UAS learning curves of Galician (using predicted and gold POS-tags, respectively), together with the values of the best single model and of the best combination (Table 7).Footnote 21
With ≈20, 000 tokens, the Galician data achieves a parsing performance of 70.68 percent/77.22 percent (pred), and 78.12 percent/82.1 percent (gold), higher values than some of those obtained in other languages (similar, however, to Brazilian Portuguese, and lower than Italian, which reaches almost 79 percent/84 percent with the same amount of training data and gold POS-tags).
Apart from that, Figures 3 and 4 also show that to beat the best single cross-lingual models we would need the following amount of Galician data: ≈3, 000 (LAS) and ≈6, 000 (UAS) tokens if we have predicted POS-tags, and ≈4, 000 (LAS) and ≈10, 000 (UAS) if we use gold POS-tags. These values follow the same tendency as mentioned for other Romance languages in Section 3.
However, the results obtained by the focused adaptation and combination of the source treebanks allow the parser to noticeably increase its performance, obtaining LAS results equivalent to the Galician parser at >16, 000 tokens, and better UAS values than using the full set of ≈20, 000. This combination achieves 70.63 percent/78.71 percent (pred) and 76.99 percent/82.44 percent (gold), but these values reach 71.45 percent/78.57 percent (pred) 78.35 percent/83.08 percent (gold) if we add the ≈20, 000 training tokens of Galician to the combined model.
The results of these evaluations show that a parser built by means of a combination of adapted treebanks is a good alternative to manually annotating a large corpus for a different language. Furthermore, a bootstrapping process with some manual revision allows researchers to evaluate the best source treebanks as well as to add these new data to improve the cross-lingual model.
Finally, we also carried out an evaluation of a parser for Galician, trained using the whole gold standard in a cross-lingual scenario: parsing all the other Romance languages. These results can be seen in Table 8, which complement those presented in Tables 3 and 4.
Cross-lingual parsing results on Romance languages using a model trained with the Galician gold standard (1,000 sentences)
Each metric row (LAS and UAS) contains both lexicalized (top) and delexicalized results (bottom). Results were obtained using predicted POS-tags. Galician values (in italic) were obtained with the 1,000 sentences splitted in 800 (train) and 200 (test), also with predicted POS-tags (≈93 percent).
Even if the training data is small (≈24, 000 tokens), and it has fewer dependency labels than the test treebanks, the results also follow the same tendency as the other Romance languages. Italian is the language on which the best results are obtained (both in LAS and UAS), followed by Portuguese and Spanish (with small differences between LAS and UAS results), and the Romanian numbers are worse than those for all the other varieties. Again, the results show that the Galician delexicalized models did not improve the parsing of any Romance language, and also that lexical distance does not seem to be a crucial factor for the cross-lingual analysis of varieties from the same linguistic family.
5 Discussion
The results of previous work on cross-lingual parsing had suggested that the good performance of this strategy on Romance languages might come from their structural similarities. Thus, in these syntactically related languages, lexical distance may play a crucial role when leveraging treebanks from one linguistic variety to another.
In this regard, the tests carried out with an adapted version of the European Portuguese treebank (automatically converted to a Spanish-like spelling, also used in Galician) seem to confirm the following hypothesis: if we approximate, in lexical terms, a source treebank to the target one, the transfer performance improves.
However, from the computed values of lexical similarity and treebank coverage (Section 3.1) together with the results of the experiments on cross-lingual parsing (Section 3.2), we can infer that lexical distance is not as important as other factors in the cross-lingual transfer of UD resources in Romance languages. Thus, the best source language for analyzing Spanish (both LAS and UAS) seems to be Italian, but this language only covers ≈2.1 percent of the Spanish dictionary (and 39.9 percent if we use the treebank-based comparison, which are the worst results on Spanish after Romanian). Moreover, European Portuguese is the variety with highest lexical coverage of Brazilian Portuguese (72 percent and 83 percent), but Spanish was the best source language for the Brazilian variety.
Concerning delexicalization, removing the tokens from the source treebanks does not improve, in most cases, cross-lingual parsing between Romance languages (in a similar way as it had been reported for three Slavic languages (Agić et al. Reference Agić, Tiedemann, Merkler, Krek, Dobrovoljc and Moze2014)). In the experiments performed, the only source language that clearly benefited from delexicalization was Romanian, whose results also followed different tendencies when compared to the other Romance languages. Furthermore, the improvements provided by lexicalized models were noticeably greater than those of the cases where the delexicalized parsers performed better. Interestingly, the impact of delexicalization is significantly correlated with the treebank lexical coverage between the source and the target varieties: the higher the coverage, the better it is to use a lexicalized parser, and vice-versa.
As pointed out in previous work (Tiedemann Reference Tiedemann2015b), the quality of the POS-tags is also critical for both cross-lingual and monolingual parsing. In our first set of experiments on Romance languages, the use of predicted POS-tags (with an average precision of ≈97 percent) involved drops of ≈1.6 percent and ≈2.4 percent in cross-lingual and monolingual parsing, respectively, if compared with parsing using gold POS-tags. In the analysis of Galician—where the predicted POS-tags only achieved ≈93 percent—, the differences were of ≈4.9 percent (cross-lingual) and ≈6.1 percent (monolingual), again compared to the results on a corpus with gold POS-tags.
The size of the training corpus does not seem to be as crucial in cross-lingual parsing as in a monolingual scenario. Large training corpora (using combinations of treebanks with more than 300k tokens) achieve equivalent performance to using between 3,000 and 7,000 tokens of the target language (which is in accordance with the UAS results presented by Lacroix et al. (Reference Lacroix, Wisniewski and Yvon2016b)), so it can be said that more data from a different language does not continuously improve the syntactic analysis. Also, the cross-lingual experiments using the Galician gold standard as source (with less than 25,000 tokens) demonstrate that even a small training corpus can perform relatively well when compared to other large datasets used for training.
Another factor that has an impact in the cross-lingual transfer of UD resources is possible divergences in the annotation of the treebanks (obviously, much lower than those that arise using other guidelines than UD). These divergences can derive (i) from linguistic phenomena and decisions of the annotators (e.g., the use of language-specific dependencies or decisions about how to label some phenomenon in a language), as well as (ii) from the properties of a treebank converted from another resource. An example of the latter can be seen in European and Brazilian Portuguese, which in their UD 1.3 version show divergences in the tokenization of contractions.
Taking the above into account and depending on the objectives of the cross-lingual parsing, some guidelines could be easily modified for improving treebank transfer (especially in terms of LAS). Moreover, the combination of different source treebanks, together with the mentioned adaptation of their annotation to the target language, noticeably improves parsing performance. This fact can be used by researchers to syntactically analyze a new language with little annotation effort.
6 Conclusions
In the present paper, we have performed an analysis of cross-lingual parsing between Romance languages using UD. The results of this analysis have served us to start the creation of a new UD treebank for Galician, by means of a previous cross-lingual parsing with reasonably high accuracy.
The experiments carried out in this work were designed to know (i) the impact that both the lexical distance between two languages and their structural similarities have in cross-lingual parsing, (ii) whether it is beneficial to perform delexicalization in a cross-lingual scenario, and (iii) to what extent we can leverage cross-lingual resources for UD parsing in Romance languages.
In this regard, the results of several evaluations suggest that—even if it is important—the lexical distance between two languages is not a key factor for cross-lingual parsing, so other properties such as syntactic differences or divergences in the annotation guidelines—or even the textual typology of the training corpus—play a crucial role in the performance of the transfer of syntactic models. Apart from that, the experiments on Romance languages have also shown that the delexicalization process is not useful in most scenarios of cross-lingual parsing, and that its impact is significantly correlated with the lexical coverage between the source and target treebanks.
After comparing the performance of transferred parsers with the learning curves of monolingual ones, we can state that a direct cross-lingual transfer using just one source treebank behaves similar to ≈3, 000 (in LAS results) and ≈7, 000 (UAS values) manually revised tokens of the target language. However, a focused combination and adaptation of different treebanks to the target language can boost these results to more than double.
Thus, the case study on the development of a new UD treebank for Galician has shown that combining and adapting resources from related languages (in our case from European Portuguese, Castilian Spanish and Italian) leads us to parse a Galician corpus with results of 76.99 percent (LAS) and 82.44 percent (UAS) without data of the target language (which would be equivalent to the use of a Galician training corpus of ≈16, 000 tokens—for similar LAS results—or >20, 000 —for UAS). Moreover, the addition of 20,000 tokens of Galician to these source treebanks increases the numbers up to 78.35 percent (LAS) and 83.08 percent (UAS), which are competitive results when compared to those obtained in other languages with large training corpora.
In sum, the experiments carried out in this paper point out that, depending on the objectives and on the available resources, it is possible to start the creation of a UD treebank (or a parser) for a new language by leveraging resources from related languages, reducing notoriously the manual effort with respect to building a treebank from scratch.
However, there are still some open questions which need further research. A more detailed analysis between varieties of the same language (such as EP and BP), and also between different treebanks of the same linguistic variety, will shed some light on the effect of syntactic differences in cross-lingual parsing. Also, extending the experiments performed in this paper to other linguistic families could also bring interesting information concerning cross-lingual transfer.
Specifically for Galician, it would be important to enlarge and improve the manual annotation of the corpus, in order to have a better view of its learning curve compared to the cross-lingual focused combinations.
In this respect, it is worth noting that this article contributes to the UD project by releasing a new treebank for Galician (Galician-TreeGal), with a manually corrected gold standard of 1,000 sentences (24,219 tokens).
