1. Introduction
In natural languages, there are many ways to express complex human thoughts and ideas. This can be achieved by exploiting compositionality, that is, concatenating simplex elements of language, and thus, yielding a more complex meaning that can be computed from the meaning of the original parts and the way they are combined. However, non-compositional phrases can also be found in languages, which are complex phrases that can be decomposed into single meaningful units, but the meaning of the whole phrase cannot (or can only partially) be computed from the meaning of its parts. Such phrases are often called multiword expressions (MWEs) and they display lexical, syntactic, semantic, pragmatic, and/or statistical idiosyncrasies (Sag et al. Reference Sag, Baldwin, Bond, Copestake and Flickinger2002; Calzolari et al. Reference Calzolari, Fillmore, Grishman, Ide, Lenci, MacLeod and Zampolli2002; Kim Reference Kim2008), which might pose problems for linguistic processing, especially in language learning and natural language processing (NLP).
From a multilingual perspective, the handling of MWEs is even more complicated. The very same meaning can be expressed by completely different words in different languages: for example, take the phrase to make a decision, which is eine Entscheidung treffen (a decision to meet) in German and döntést hoz (decision-ACC bring) in Hungarian, that is, three different verbs are used here whose original meanings are quite distinct. Hence, MWEs need special attention across languages as well.
In this paper, we focus on a specific type of MWEs, namely, light verb constructions (LVCs). They are verb and noun combinations in which the verb has lost its meaning to some degreeFootnote a and the noun is used in one of its original senses, most typically denoting an event or state (e.g., have lunch or have a walk). They usually share their syntactic structures with those of compositional syntactic phrases (have a walk vs. have a dog), which may impede their automatic identification. However, in several NLP applications like information retrieval or machine translation it is important to identify LVCs in context since they require special care, especially because of their semantic features; however, the detection of LVCs is often hindered by the lack of annotated resources of sufficient size. Here, we will overcome these difficulties by showing that annotated data from other languagesFootnote b can be fruitfully exploited in training supervised machine learning methods to detect LVCs in texts.
We will first analyze the characteristics of LVCs with special regard to four different languages (English, German, Spanish, and Hungarian) from a linguistic point of view based on parallel corpus data. Then, we will provide a standardized (i.e., language-independent) representation of LVCs that can be used in machine learning experiments. After, we will demonstrate that additional language data can be successfully employed in improving the performance of supervised LVC detection for a given language.
The main contributions of this study can be summarized as follows:
• We provide an extended definition of LVCs and a linguistic characterization of LVCs in English, German, Spanish, and Hungarian on the basis of data from the 4FX corpus (Rácz, Nagy T., and Vincze Reference Rácz, Nagy, Vincze, Calzolari, Choukri, Declerck, Loftsson, Maegaard, Mariani, Moreno, Odijk and Piperidis2014).
• We offer a standardized linguistic representation of LVCs across languages with morphological, syntactic, and lexical features, which is deeply rooted in theoretical linguistics but we bear in mind its applicability in machine learning experiments too.
• We introduce our machine learning-based method to automatically identify all LVC occurrences in raw text in four different languages. Our machine learning method to automatically detect LVCs has already been implemented for English and Hungarian (Vincze, Nagy T., and Farkas Reference Vincze, Nagy and Farkas2013a), but here we adapt this method to German and Spanish as well.
• As a first step of our machine learning method, we provide a data-driven candidate extraction method for all the different languages, which is based on the above-mentioned standardized syntactic relations.
• As a second step of our machine learning method, we rely on the classification approach reported in Vincze et al. (Reference Vincze, Nagy and Farkas2013a), but we extend the feature set with some new language-specific features too.
• We report results for each language separately concerning LVC detection with this machine learning method.
• We also examine how the different languages can affect each other, therefore, we carry out some cross-language experiments for each possible combination of applying a language as target language and a subset or union of the other three languages as source language.
• Here, we apply a domain adaptation-based cross-language adaptation technique and examine how (domain) adaptation can enhance the results if we only have a limited amount of annotated target language data.
• We also examine the effectiveness of feature types by applying an ablation analysis in several machine learning settings.
• As in the case of English and Hungarian, we have access to manually annotated data from other domains, we also investigate the effect of simple domain adaptation techniques to reduce the gap between domains.
• We combine domain adaptation techniques with language adaptation techniques in the case of English and Hungarian and show that both out-domain and additional language data can improve performance.
The remainder of the paper is structured as follows: In Section 2, we examine the analyzed languages and their LVCs. In Section 3, we summarize related work, where we present the related corpora and discuss previous methods for identifying LVCs. Then, we discuss the 4FX parallel corpus, the annotation principles and the statistics on corpus data, which is followed by Section 5, where we present our method for the automatic detection of LVCs on different languages. Here, we introduce our approaches to cross-language adaptation and domain adaptation with additional language data too. The results are presented in detail in Section 6. Next, results are elaborated on in Section 7. Lastly, the contributions of the paper are summarized and some possible directions for future study are briefly mentioned.
2. Characteristics of LVCs
It has been earlier pointed out that especially in computational linguistics, different authors apply different names for the same phenomena or rather, they call different phenomena by the same name (see the example of hedges and speculation, discussed in Szarvas et al. Reference Szarvas, Vincze, Farkas, Móra and Gurevych2012), making it difficult to compare results obtained by different authors on different datasets. The same is true for LVCs (both as a term and as a linguistic phenomenon). Thus, we think it necessary to give a short summary of how these verb + noun constructions are accounted for in the (computational) linguistic literature.
Traditionally, light verbs have a (direct or indirect) object which is a predicative noun denoting an eventuality. On the other hand, Krenn (2008) deals with just verbs + prepositional phrases. German theoretical linguists (see, e.g., Duden 2006) usually do not consider verb + subject combinations as Funktionsverbgefüge (a German term for fixed verb + noun constructions) but Hungarian linguists do (Vincze Reference Vincze2011). However, Mel’čuk (Mel’čuk Reference Mel’čuk1974; Mel’čuk, Clas, and Polguére Reference Mel’čuk, Clas and Polguère1995; Mel’čuk Reference Mel’čuk2005), among others, extended the definition to cover cases where the noun is the subject or an oblique/prepositional argument of the verb. Verbs that bear aspectual information may also occur in this group (Danlos Reference Danlos, Nakamura, Laporte, Dister and Fairon2010; Vincze Reference Vincze2011; Varga Reference Varga and Kakoyianni-Doa2014).
As for their semantics, some verbal components are semanticallymore bleached than the others (Alonso Ramos Reference Alonso Ramos2004; Sanromán Vilas Reference Sanromán Vilas2009). Meyers, Reeves, and Macleod (Reference Meyers, Reeves and Macleod2004) distinguished support verbs and light verbs: semantically empty support verbs are called light verbs, that is, the term support verb is a hypernym of light verb. Kearns (Reference Kearns2002) also made a distinction between two subtypes of what is traditionally called LVCs. True LVCs such as to give a wipe or to have a laugh and vague action verbs such as to make an agreement or to do the ironing differ in some syntactic and semantic properties and can be separated by various tests, for example, passivization, WH-movement, pronominalization, etc.
Thus, a wide range of fixed verb + noun constructions with peculiar semantic behavior have received intensive attention in many languages (for definitions, see, e.g., Langer Reference Langer, Langer and Schnorbusch2005, Mel’čuk Reference Mel’čuk2005, and Danlos Reference Danlos, Nakamura, Laporte, Dister and Fairon2010). Here, we summarize their most important general characteristics:
• they are lexically fixed expressions;
• they follow the base form: verb prep? det? noun or prep? det? noun verb (using the ? operator as in regular expressions);
• their meaning is not totally compositional;
• the noun typically refers to an action or event;
the noun is a syntactic argument of the verb;
the verb contributes to the meaning only to a small degree (e.g., aspectual information).
In this paper, we take a comprehensive approach and use the term light verb construction as widely as possible. That is, we do not make any morphosyntactic, lexical, semantic, and language-specific restrictions (see also Section 3.3), and we consider each verb + (preposition) + (article) + noun combinationFootnote c as a LVC that fulfills the above criteria. Prepositions and determiners may be also part of the construction (e.g., take into account), as we do not restrict ourselves to the investigation of only verb + noun constructions. We believe that using an extended definition of LVCs is beneficial for computational linguistic applications as a wider range of constructions that require special treatment can be identified in this way.
In the following, we will focus on the four languages we are interested in, namely, English, German, Spanish, and Hungarian, and we will also briefly present their grammatical characteristics in general and describe the main characteristics of LVCs as well.
2.1 English
English is known as a prototypical configurational language, that is, it has strict word order and poor morphology. The syntactic roles of arguments are mostly determined by their position within the sentence. As for LVCs, the canonical order (i.e., the form that occurs in dictionaries) is verb + noun, but there can also be an article and/or a preposition in between and due to passivization, the two components may not be adjacent.
Earlier, several studies were carried out to investigate the syntactic features of English LVCs (Hale and Keyser Reference Hale and Keyser2002; Meyers et al. Reference Meyers, Reeves and Macleod2004). They usually share their syntactic structures with productive or literal verb + noun constructions on the one hand and idiomatic verb + noun expressions on the other, see, for example, Fazly and Stevenson (2007), as reflected in the examples make a cake (productive), make a decision (LVC), and make a meal (idiom) (Vincze, Nagy T., and Zsibrita Reference Vincze, Nagy and Zsibrita2013b). Hence, their identification cannot be based on solely syntactic patterns. Kearns (Reference Kearns2002) distinguished between two subtypes of what is traditionally called LVCs: true LVCs and vague action verbs differ in some syntactic and semantic properties and can be separated by various tests, like passivization, WH-movement, and pronominalization. Vincze et al. (Reference Vincze, Nagy and Zsibrita2013b) classify LVCs from morphological and semantic aspects, and they also take into account theoretical and computational linguistic considerations.
2.2 German
German forms part of the Germanic language family and it has a morphologically complex grammar system. Being a flective language, grammatical functions are expressed mostly by the alternation of word forms. As for verbs, they are conjugated for person, number, tense, mood, and voice, and in nominal declination, the number, gender, and case can be specified (Bußmann Reference Bußmann2002).
The canonical form of German LVCs is a noun (eventually preceded by a preposition and/or an article) + infinitive verb (Fleischer, Helbig, and Lerchner Reference Fleischer, Helbig and Lerchner2001; Duden 2006). The verb can be conjugated thus can appear in many surface forms, besides, due to word order reasons, it may precede or follow the noun.
LVCs have been studied by German theoretical linguistic research for over 50 years now (Daniels Reference Daniels1963; Häusermann Reference Häusermann1977). However, as Heine demonstrates, in descriptive grammars, there is still no consensus about the main features of LVCs (Heine Reference Heine2006). In Duden, prototypical LVCs are characterized by features like the determined use of article, fixed way of negation (i.e., it is lexically determined whether the given phrase can be negated with the determiner kein or with the adverb nicht), or no free choice of attributes in the case of the nominal component (Duden 2006). In certain cases, they can be formed with the copula sein “to be” and bleiben “to stay” as well (Duden 2006) and some verb–noun combinations in genitive case also count as LVCs if they express an opinion (Helbig and Buscha Reference Helbig and Buscha2001).
2.3 Spanish
Spanish belongs to the Ibero-Romance languages, where verbs can be conjugated for person, number, tense, mood, and aspect and nouns, adjectives, and determiners also have an inflectional paradigm since they carry information about number and gender (RAE 2009).
The canonical form of Spanish LVCs is an infinitive + preposition? + article? + noun, hence, their canonical word order is just like in English. In some cases, however, these elements can also be separated, yielding non-canonical, non-adjacent forms.
Until recently, there were only few studies about Spanish LVCs, but nowadays they have come to the fore in the NLP community (Alonso Ramos Reference Alonso Ramos2000). The main features of Spanish LVCs are described among others in Ramos (2004), Blanco Escoda (Reference Blanco Escoda2000), and Bosque (Reference Bosque, Zagona, Maléln and Herschenson2001). In the field of applied linguistics, we would like to mention Buckingham’s work, in which a corpus of written academic language is used in order to identify the most common LVCs (Buckingham Reference Buckingham2009). The study of Leoni de León (Reference de León, Nolan and Periñán-Pascual2014) is also worth noting, as it gives an overview about the lexical status and syntactic features of MWEs and describes a system for their identification.
2.4 Hungarian
Hungarian is an agglutinative language, which means that a word can have hundreds of word forms due to inflectional or derivational morphology (É. Kiss Reference é. Kiss2002). Hungarian word order is related to information structure, thus, the position relative to the verb has no predictive force as regards the syntactic function of the given argument. In contrast, the grammatical function of words is determined by case suffixes.
The canonical form of a Hungarian LVC is a bare noun + third person singular verb, where the noun can function not only as the object of the verb but also another argument as well. However, the verb may precede the noun, or they may not be adjacent for word order reasons, moreover, the verb may occur in different surface forms inflected for tense, mood, person, and number.
Varga (Reference Varga and Kakoyianni-Doa2014) analyzed Hungarian LVCs of the form verb + object in the lexicon-grammar framework. Vincze et al. (Reference Vincze, Nagy and Zsibrita2013b) provided a detailed analysis of English and Hungarian LVCs from a morphological viewpoint. Hungarian LVCs are described in detail from a syntactic and semantic viewpoint in Vincze (Reference Vincze2011), and they are also contrasted with English as well.
3. Related work
Next, we will present the related corpora and discuss previous methods used to identify LVCs. We mostly focus on databases and methods developed for our languages investigated in this paper, however, we would like to emphasize that there is an extensive ongoing work on MWEs in other languages, for example, within the PARSEME international research network.Footnote d
3.1 Related corpora
In order to build supervised machine learning-based methods to automatically identify LVCs in texts, well-designed and tagged corpora are invaluable for training and testing statistical models. Here, we present the most commonly used databases and corpora annotated for LVCs in several languages.
Krenn (2008) developed a database of German PP-verb combinations, while Kolesnikova and Gelbukh (Reference Kolesnikova and Gelbukh2010) collected and classified Spanish verb–noun combinations. Vincze et al. (Reference Vincze, Nagy and Berend2011b) manually annotated several types of MWEs (including LVCs) in English Wikipedia texts, while Tu and Roth (Reference Tu and Roth2011) also published an English corpus, where true LVCs were manually marked. PropBank also contains annotation for LVCs (Hwang et al. Reference Hwang, Bhatia, Bonial, Mansouri, Vaidya, Xue and Palmer2010). Hungarian LVCs weremanually marked in the Szeged Treebank (Vincze and Csirik Reference Vincze and Csirik2010). An English–Hungarian annotated parallel corpus of LVCs was recently published (Vincze Reference Vincze2012). Moreover, English, German, Spanish, and Hungarian legal texts were recently annotated for LVCs in a parallel corpus (Rácz et al. Reference Rácz, Nagy, Vincze, Calzolari, Choukri, Declerck, Loftsson, Maegaard, Mariani, Moreno, Odijk and Piperidis2014). As this corpus will be employed in most of our experiments, it will be described later on in Section 4.
Due to the PARSEME Shared Task that aimed at identifying verbal MWEs in raw texts, a new multilingual corpus has been published (Savary et al. Reference Savary, Ramisch, Cordeiro, Sangati, Vincze, QasemiZadeh, Candito, Cap, Giouli, Stoyanova and Doucet2017; Ramisch et al. Reference Ramisch, Cordeiro, Savary, Vincze, Barbu Mititelu, Bhatia, Buljan, Candito, Gantar, Giouli, Güngör, Hawwari, Iñurrieta, Kovalevskaitė, Krek, Lichte, Liebeskind, Monti, Parra Escartín, QasemiZadeh, Ramisch, Schneider, Stoyanova, Vaidya and Walsh2018) in 2017 and 2018. This corpus contains texts for 18 different languages, all annotated for each occurrence of idioms, LVCs, and other types of verbal MWEs. The organizers of the shared task provided a standardized guideline for annotation, which was followed for all languages. Thus, the data in the different languages are easily comparable and provide space for multilingual experiments. However, we did not choose to experiment on this dataset as we first wanted to test our approach on a parallel corpus. In this case, differences in the results obtained cannot be attributed to differences in domain, genre, and the available quantity of annotated texts for different languages. If our method seems to be feasible on the 4FX dataset, it can be extended to other corpora in a future study.
3.2 Methods for identifying LVCs
Several applications have been developed for identifying LVCs in running texts. The methods applied may be divided into three categories, namely, statistical, rule-based, and machine learning-based approaches. It should be noted, however, that most of these studies applied restrictions as regards the set of LVCs to be identified (see Section 3.3 below), hence, their results are not reported here as they are not directly comparable either to each other or to our results.
Stevenson, Fazly, and North (Reference Stevenson, Fazly and North2004) and Fazly and Stevenson (Reference Fazly and Stevenson2007) built LVC detection systems with statistical features. Stevenson et al. (Reference Stevenson, Fazly and North2004) focused on classifying true LVC candidates containing the verbs make and take. Fazly and Stevenson (Reference Fazly and Stevenson2007) used linguistically motivated statistical measures to distinguish subtypes of verb + noun combinations.
Nagy T., Vincze, and Berend (Reference Nagy and Vincze2011), Vincze, Nagy T., and Berend (Reference Vincze, Nagy and Berend2011a), and Nagy T. and Vincze (Reference Nagy and Vincze2011) applied rule-based systems to automatically detect LVCs in running texts. Vincze et al. (Reference Vincze, Nagy and Berend2011a) exploited shallow morphological features to identify LVCs in English texts, while the domain specificity of the problem was highlighted in Nagy T. et al. (Reference Nagy and Vincze2011). Nagy T. and Vincze (Reference Nagy and Vincze2011) identified verb–particle constructions and LVCs in running texts using rule-based methods and examined how adding certain features could affect the overall performance.
Tan, Kan, and Cui (Reference Tan, -Y and Cui2006) and Tu and Roth (Reference Tu and Roth2011) attempted to identify true LVCs by applying machine learning techniques. Tan et al. (Reference Tan, -Y and Cui2006) combined statistical and linguistic features and trained a random forest classifier to separate LVC candidates, while Tu and Roth (Reference Tu and Roth2011) trained a Support Vector Machine with contextual and statistical features on their balanced dataset annotated for English LVCs. Bannard (Reference Bannard2007) sought to identify verb and noun constructions in English on the basis of syntactic fixedness. Nagy T., Vincze, and Farkas (Reference Nagy, Vincze and Farkas2013) focused on the full coverage identification of English LVCs in running texts. They classified LVC candidates using a decision tree algorithm, which was based on a rich feature set with new features like semantic and morphological features.
In the PARSEME Shared Tasks, several machine learning-based methods competed on the four languages under investigation, among others. Some of them relied on parsing (Al Saied, Constant, and Candito Reference Al Saied, Constant and Candito2017; Nerima, Foufi, and Wehrli Reference Nerima, Foufi and Wehrli2017; Simkó, Kovács, and Vincze Reference Simkó, Kovács and Vincze2017; Waszczuk Reference Waszczuk2018), whereas others exploited sequence labeling using CRFs (Boroş et al. Reference Boroş, Pipa, Barbu Mititelu and Tufiş2017; Maldonado et al. 2017; Moreau et al. Reference Moreau, Alsulaimani, Maldonado and Vogel2018) and neural networks (Klyueva, Doucet, and Straka Reference Klyueva, Doucet and Straka2017; Berk et al. Reference Berk, Erden and Güngör2018; Boroş and Burtica Reference Boroş and Burtica2018; Ehren, Lichte, and Samih Reference Ehren, Lichte and Samih2018; Stodden, QasemiZadeh, and Kallmeyer Reference Stodden, QasemiZadeh and Kallmeyer2018; Zampieri et al. Reference Zampieri, Scholivet, Ramisch and Favre2018).
3.3 Restrictions on LVCs
As earlier studies applied some restrictions on LVCs to be detected, here we will classify them according to the restrictions they utilize.
Some methods previously used (Tu and Roth Reference Tu and Roth2011; Fazly and Stevenson Reference Fazly and Stevenson2007) employed morphosyntactic restrictions and just treated the verb–object pairs as potential LVCs (Stevenson et al. (Stevenson et al. Reference Stevenson, Fazly and North2004; Tan et al. Reference Tan, -Y and Cui2006). English verb + prepositional constructions were mostly neglected in previous studies, although the annotated corpora mentioned in Section 3.1 contained several examples of such structures, for example, take into consideration or come into contact. Some researchers applied lexical restrictions and filtered LVC candidates by selecting only certain verbs that may be part of the construction. One such example is Tu and Roth (2011), where the authors chose six light verbs (make, take, have, give, do, get) to focus on. Furthermore, semantic restrictions were applied by some earlier researchers, so they just focused on the identification of true LVCs, neglecting vague action verbs (Stevenson et al. Reference Stevenson, Fazly and North2004; Tan et al. Reference Tan, -Y and Cui2006; Tu and Roth Reference Tu and Roth2011).
Here,—in accordance with Nagy T. et al. (2013)—we seek to identify each type of LVCs and do not restrict ourselves to certain types of LVCs as we think that they all are relevant for NLP (e.g., they must be treated as one complex predicate (Vincze Reference Vincze2012)).
4. The corpus
Now, we will describe the 4FX corpus (Rácz et al. Reference Rácz, Nagy, Vincze, Calzolari, Choukri, Declerck, Loftsson, Maegaard, Mariani, Moreno, Odijk and Piperidis2014) in detail and analyze language-specific features on the characteristics and distribution of LVCs based on empirical data from the corpus.
4.1 Statistical data on the corpus
Texts from our 4FX corpus are based on the JRC-Acquis Multilingual Parallel Corpus, consisting of legal texts for a range of languages used in the European Union (Steinberger et al. Reference Steinberger, Pouliquen, Widiger, Ignat, Erjavec and Tufiş2006). For an earlier investigation on LVC detection (Vincze et al. Reference Vincze, Nagy and Zsibrita2013b), 60 documents from the English version of the corpus were randomly selected and LVCs in them were annotated. We annotated the Spanish, German, and Hungarian equivalents of those 60 documents, thus, yielding a quadrilingual parallel corpus named 4FX. The annotation was carried out by two native speakers of Hungarian who have an advanced level knowledge of English, German, and Spanish. Data on annotated texts can be seen in Table 1.
Statistical data on the 4FX parallel corpus
4.2 Annotation principles
In order to make the annotation in the different languages as uniform as possible, we adapted the guidelines applied while constructing the SzegedParalellFX corpus.
During annotation, we decided to mark only the verb that belonged to the LVC (i.e., passive or perfect auxiliaries were not marked as part of an LVC) but verbal prefixes that were separated from their main verb due to word order reasons were marked in German and Hungarian.
We should also mention that we marked not only the prototypical constructions (VERB, e.g., make contracts), but also their nominal (marked as NOM, e.g., making of contracts) and participial forms (PART, e.g., contracts made) were included in the annotation. Constructions marked with PART and NOM were marked regardless of whether they consisted of one or more elements (cf. szerzödéskötés “making a contract” or Durchführung einer Untersuchung “carrying out an investigation” ). Also, we took into account the non-adjacent forms of LVCs (SPLIT), like in contractsthat have been made. The corpus contains annotation for verbal, participial, nominal, and split occurrences of LVCs as well. However, we will neglect participial constructions spelt as one word and also the nominal occurrences of LVCs in our experiments since their identification would require methods radically different from syntax-based methods applicable to verbal and non-compound participial forms.
4.3 Analysis of corpus data
A comparison of the data on the four languages reveals some interesting facts. The statistics on the subtypes of LVCs presented in Table 2 displays intriguing tendencies if we take into account the fact that the parallel versions of the same corpus were the basis of our manual annotation. As can be seen, Spanish constructions marked as VERB occurred more than twice as frequently as those in English. This fact accords with our belief that there is a fundamental difference among LVC types of the languages as well as among the languages regarding their LVCs.
One of the most salient results is that, in German, the number of non-adjacent, SPLIT LVCs exceeds the frequency of all the rest of languages. This may be attributed to the strict rules of German word order. The verb has to occupy the first or second place in a main clause, but its arguments have a more flexible position, for example,
Diese Verordnung tritt am 31. März 2006 in Kraft.
this-FEM directive step-3SG on-MASC-DAT 31 March 2006 in force
“This Directive shall enter into force on the 31 March 2006.”
Types of LVCs in the 4FX corpus
NOM, nominal LVCs; VERB, verbal occurrences; SPLIT, split LVCs; PART, participial LVCs.
Yet another peculiarity of this language is the outstanding frequency of nominal constructions, which is noticeable in Hungarian to some extent as well. This feature is most probably due to the so-called Nominalstil, the preferred use of nominal forms in technical jargon. Consequently, our results seem to confirm statistically a fact often referred to in related German literature too, namely, that legal texts abound in nominalized constructions and LVCs in general (Duden 2006). Concerning this feature, German and Hungarian are quite close to each other. In the latter case, however, the wide-ranging rules of word building make it possible to create three different participial forms as well, which are less frequently used in the other languages. This fact might explain the dominance of Hungarian PART constructions in the corpus.
As for a qualitative analysis of the data, Table 3 shows that there are some common LVCs that are frequent in each of the four languages, for instance:
to enter into force—in Kraft treten—entrar en vigor—hatályba lép
to grant aid—Hilfe gewähren—conceder ayuda—támogatást nyújt.
LVC occurrences in the 4FX corpus
Light verb occurrences in the 4FX corpus
Statistics on light verb occurrences in the 4FX corpus
Some other constructions are found in the top five LVCs in three of the languages (but not in German), such as
to take into account—tener en cuenta—figyelembe vesz
to receive aid—beneficiarse de una ayuda—támogatásban részesül.
These constructions are the most frequent in the corpus and they are also characteristic of the legal language. On the other hand, we also found language-specific LVCs in the 4FX corpus that do not have an equivalent in all or any of the other languages, just like the English phrase having regard to corresponds to the Hungarian phrase tekintettel regard-INS “with regard to.”
Next, as is seen in Table 5, concerning the number of LVC lemmas, Spanish comes first but concerning the sum of light verbs, German comes top. With regard to these values, English comes quite low, which suggests that LVCs here are less diverse than in the other languages, at least in the legal domain. Analyzing the number of hapax LVCs (i.e., those that occur only once in the dataset), a similar picture is seen, since the frequency of LVCs with only one occurrence in the corpus is the lowest in English.
5. Automatic detection of LVCs
In order to see how the different languages can affect each other, an implementation of the language-independent representation of LVCs is required. Later, we can apply a domain adaptation-based cross-language adaptation technique and discover how (domain) adaptation can enhance the results if we only have a limited amount of annotated target data.
As mentioned in Section 3.1, different types of texts may contain different types of LVCs. Therefore, we also investigate the effect of simple domain adaptation techniques to reduce the gap between the legal text domain and other domains, as there are other manually annotated corpora available for English and Hungarian, where LVCs are manually marked.
5.1 Candidate extraction
As LVCs were manually annotated in the 4FX parallel corpus in four different languages, we were able to define a language-independent syntax-based candidate extraction method to automatically extract potential LVCs from raw text, based on Nagy T. et al. (2013). Hence, we examined the syntactic relations among the LVCs’ verbal and nominal components in the four different languages. To parse the English, German, and Spanish parts of the 4FX corpus, the Bohnet parser (Bohnet Reference Bohnet2010) was applied, which was trained on the English part of the CoNLL shared task (Surdeanu et al. Reference Surdeanu, Johansson, Meyers, Màrquez and Nivre2008), on the TIGER treebank (Brants et al. Reference Brants, Dipper, Eisenberg, Hansen-Schirra, König, Lezius, Rohrer, Smith and Uszkoreit2004) in German and on the IULA treebank (Marimon et al. Reference Marimon, Fisas, Bel, Arias, Vázquez, Vivaldi, Torner, Villegas and Lorente2012) in Spanish. In the case of Hungarian, the state-of-the art Hungarian dependency parser, magyarlanc (Zsibrita, Vincze, and Farkas Reference Zsibrita, Vincze and Farkas2013), was used, which was trained on the Szeged Dependency Treebank (Vincze et al. Reference Vincze, Szauter, Almási, Móra, Alexin and Csirik2010).
As the different models on different languages were trained on different treebanks, the parsed texts have different dependency representations. For instance, the verb–direct object dependency relation is presented by dobj in English, OA in German, DO in Spanish, and OBJ in Hungarian. Hence, it was necessary to standardize the different representations of the various syntactic relations and apply the same dependency label for the same grammatical relation across the languages.Footnote e Table 6 shows the distribution of standardized dependency label types on the four languages in the 4FX parallel corpus. In the candidate extraction phase, we treated as a potential LVC each verb and noun combination, which are connected with verb–direct object (dobj), the verb–(passive) subject (subj/nsubjpass), the verb–adpositional (pobj), and noun–participial modifier (partmod) syntactic relations.
Syntactic relations of LVCs annotated in the 4FX parallel corpus
dobj, direct object; pobj, adpositional object; nsubjpass, subject of passive verb; partmod, participial modifier; other, a syntactic relation other than the above; none, no syntactic relation between the noun and the verb within the LVC.
5.2 Standardized feature representation on different languages
For the automatic classification of the candidate LVCs a machine learning-based approach (Vincze et al. Reference Vincze, Nagy and Farkas2013a) was utilized, which was successfully applied to English and Hungarian. This method is based on a rich feature set with the following categories: statistical, lexical, morphological, syntactic, and orthographic. In order to detect Spanish and German LVCs in text, we also implemented this feature set in both new languages and we defined some new language-specific features too. A standardized representation of the feature set is also required, as we also would like to investigate how the different languages influence each other. Therefore, the same features in different languages were associated with each other. This feature set includes language-independent and language-specific features as well. Mainly the language-independent features aim to grab general features of LVCs while language-specific features can be applied due to the different grammatical characteristics of the four languages. Table 7 shows which features were applied to which language and a detailed description of the features follows below.
Morphological features: As the nominal component of LVCs is typically derived from a verbal stem (make a decision) or coincides with a verb (have a walk), the VerbalStem binary feature focuses on the stem of the noun; if it had a verbal nature, the candidates were marked as true. The POS-pattern feature examines the part-of-speech tag sequence of the potential LVC. If it matched one pattern typical of LVCs (e.g., verb + noun) the candidate was marked as true; otherwise as false.
The basic feature set and language-specific features
Here, we also defined some language-specific features. In the case of English, the morphological analyzer we applied was able to distinguish between main verbs and auxiliary verbs, therefore, we defined a feature for do and have to denote whether or not they were auxiliary verbs in a given sentence as they often occur as light verbs as well. As Hungarian is a morphologically rich language, we were able to define various morphology-based features. For this, we basically used the morphological analyses of words. In this way, we defined some agglutinative features like the mood and the type of the verbs, the case, the number of possessor, the person of the possessor, and the number of the possessed of the noun. Nouns which were historically derived from verbs but were not treated as derivations by the Hungarian morphological parser were also added as a feature. The gender of nouns was also utilized as feature in the case of German and Spanish, where the nominal components of LVCs often belong to the feminine gender, due to their derivational suffixes. Moreover, we also defined a German-specific feature which checked if the nominal part of the candidate was a compound noun or not. We also defined a special feature for encoding Spanish participles as the morphological analysis did not make any distinction between adjectives and participles, and while participles can be parts of LVCs, adjectives cannot be.
Orthographic features: The suffix feature is also based on the fact that many nominal components in LVCs are derived from verbs. This feature checks if the lemma of the noun ends in a given character bi- or trigram. We defined these specific bi- and trigrams for all of the four languages. The number of words of the candidate LVC was also noted and applied as a feature.
Statistical features: Potential English LVCs and their occurrences were collected from 10,000 English Wikipedia pages by the candidate extraction method. The number of occurrences was used as a feature in case the candidate instantiated one of the four syntactic relations identified in Section 5.1.
Lexical features: Here, we exploit the fact that the most common verbs are typically light verbs. Therefore, 15 typical light verbs were selected for all of the four languages (cf. Table 4) and we investigated whether the lemmatized verbal component of the candidate was one of these 15 verbs. Due to the language differences, the typical light verbs in various languages are not the same. But for the standardized feature representation, a uniform treatment of these verbs was required. Therefore, we unified the lists of the most common light verbs in all of the four languages and translated them to all the languages. For instance, if make was among the most frequent English light verbs, then its German, Spanish, and Hungarian counterparts were added to the list as well, regardless of whether they were among the most frequent light verbs in the specific language. In this way, quadruples like make—machen—hacer—tesz were formed. The list got in this way contains (four times) 29 verbs. The lemma of the noun was also applied as a lexical feature. The nouns found in LVCs were collected from the treebanks on which the parsers were trained.
Moreover, lists of lemmatized LVCs were also applied as a feature. In the case of English, manually annotated LVCs were collected and lemmatized from the English part of the SzegedParalellFX (Vincze Reference Vincze2012), while the filtered version of the German PP-Verb Collocations list (Krenn Reference Krenn2008) was applied in German. The filtered version of Spanish verb–noun lexical function dictionary (Kolesnikova and Gelbukh Reference Kolesnikova and Gelbukh2010) was utilized in Spanish and the manually annotated LVCs from the Hungarian part of the above-mentioned parallel corpus were collected for Hungarian. In German and in Spanish, filtering involved the selection of items that conformed to our definition of LVCs. These lists were also used in our baseline experiments.
Syntactic features: As the candidate extraction methods essentially depended on the dependency relation between the nominal and the verbal part of the candidate, they could also be utilized in identifying LVCs. Here, the dependency labels which were used in candidate extraction and introduced in Section 5.1 were defined as features. If the noun had a determiner in the candidate LVC, it was also encoded as another syntactic feature.
5.3 Machine learning-based candidate classification
As Nagy T. et al. (2013) found that decision trees performed the best in this task, we also applied this machine learning algorithm. Therefore, the J48 classifier of the WEKA package (Hall et al. Reference Hall, Frank, Holmes, Pfahringer, Reutemann and Witten2009) was trained on the above-mentioned feature set, which implements the C4.5 (Quinlan Reference Quinlan1993) decision tree algorithm.
As the different parts of the 4FX corpus were not big enough to split them into training and test sets of appropriate size, we evaluated our models in a 10-fold cross-validation manner at the instance level on each part of the corpus. However, in the 4FX corpus, only LVCs were marked in the text, that is, no negative examples were annotated. Therefore, all potential LVCs were treated as negative which were extracted by the candidate extraction method but were not marked as positive in the gold standard. Thus, in the dataset negative examples were overrepresented.
As Table 6 shows, the syntax-based candidate extraction methods did not cover all manually annotated LVCs in the different parts on the 4FX corpus. Hence, we treated the omitted LVCs as false negatives in our evaluation.
As a baseline, a context-free dictionary lookup method was applied in the four languages. Here, we applied the same LVC lists which were described among the lexical features in Section 5.2. Then, we marked candidates of the syntax-based candidate extraction methods as LVCs if they were found in the list.
In order to examine the effectiveness of each individual feature type, we carried out an ablation analysis in the four different languages. Tables 12–15 tell us how useful the individual features proved to be for the four languages.
5.4 Language adaptation
Here, we introduce our language adaptation technique, which is very similar to domain adaptation. In most cases, domain adaptation can enhance the results if we only have a limited amount of annotated target data. When we only have a limited set of annotated data from one domain, but there are a plenty of data available in another domain, we can apply domain adaptation methods to achieve better results on the target domain. Using the domain with a lot of annotated data as the source domain and a domain with limited data as the target domain, domain adaptation techniques can successfully contribute to the learning of a model for the target domain.
Here, we treated the different parts of the 4FX corpus with different languages as different domains. Furthermore, the above-mentioned standardized feature representation of LVCs on different languages allowed us to focus on the portability of models trained on different parts of the 4FX corpus. We were also able to investigate the effect of the language adaptation techniques so as to reduce the gap between the language pairs.
Therefore, we were able to apply the adapted machine learning-based method with language-specific features to each language to examine how much the different language models affected each other.
In the pure in-language setting, a 10-fold cross-validation was performed at the candidate level on each subcorpus of the 4FX corpus. To compare the different languages, a pure cross-language setting (CROSS) was utilized, where our model was trained on the source language and evaluated on the target (i.e., no labeled target language datasets were used for training); for example, we trained the model on the English part of corpus and tested it on the Hungarian part.
For the cross-experiments, a unified representation of the candidate LVCs on different languages was required. However, as Table 7 shows, language-specific features were also defined in each language. Therefore, the basic feature set was extended with all of the language-specific features on each language.
A very simple approach was used for language adaptation (ADAPT): we applied 10-fold cross-validation and for each fold, we used 90% of the target language as training data and the other 10% was held out for test. The source language training dataset was extended with instances from the target language training dataset. First, 10% of the target language training dataset was added to the source language training dataset, and then we kept adding 25%, 50%, and finally all of the target training instances. As the source language, we applied all possible combinations of the non-target languages, that is, we added only just one of the languages, two languages, or all the three languages.
Experimental results when the English part of the corpus was used as target and source of different language pairs in terms of precision, recall, and F-score
EN, English; DE, German; ES, Spanish; HU, Hungarian; ADAPT, language adaptation setting; CROSS: cross-language setting.
To evaluate language adaptation, our machine learning-based method was trained on the union of the source data and the candidates selected from the target language and we performed a 10-fold cross-validation at the candidate level. Each time, 90% of target data was utilized for training and the other instances were held out for testing.
The results of the experiments on language adaptation can be seen in Table 8 for English, Table 9 for German, Table 10 for Spanish, and Table 11 for Hungarian. The highest values are marked with bold.
5.5 Domain adaptation with additional language data
Here, we also investigated how we could reduce the distance between the legal domain and other domains if only limited annotated data were available for the other domains and we also exploited additional language data here. We applied a similar domain adaptation technique as in the case of the language adaptation presented in Section 5.4. We report results only for English and Hungarian, as there are other manually annotated corpora available where just English and Hungarian LVCs are manually marked.
In the case of English, the Wiki50 Corpus (Vincze et al. Reference Vincze, Nagy and Berend2011b) was applied as the English target domain and the German and the English parts of the 4FX corpus were employed as the source domain as it proved the best combination during the language adaptation setting in Table 8. On the other hand, the Hungarian target domain was the short business news domain of the Szeged Corpus (Vincze and Csirik Reference Vincze and Csirik2010). Based on the language adaptation results in Table 11, the union of the English and Hungarian parts of the 4FX corpus was selected as the source. In both settings, data from the 4FX corpus were always used as out-domain source data.
Here, we also examined how language adaptation could help the overall performance on the other domain. Therefore, we applied a two-step training. First, our model was trained on the union of the whole set of the additional language data of the 4FX corpus and the randomly selected instances of the in-language data from the 4FX corpus. First, we extended the training dataset with 10% target instances, than kept adding 25%, 50%, and 100% of the target instances. Then, the remaining part of the in-language but out-domain data was treated as test set, and we evaluated our model on it, to produce an automatically labeled (silver standard) dataset. The union of the automatically labeled test set (including in-language but out-domain data) and the additional language data (with gold standard annotation) were applied as the training dataset where our model was trained in the second step. Afterwards, this model was evaluated on the target dataset in a 10-fold cross-validation at the instance level.
System architecture of the domain adaptation with additional language data.
Later on, the training dataset was extended with instances from the target domain. As we applied 10-fold cross-validation, we used 90% of target data for training for each fold and additional instances were held out for testing. Similar to the language adaptation setting, we kept adding 10%, 25%, 50%, and finally all of the training instances from the target domain, and our model was trained on the union of the additional language data, selected instances from the out-domain data and in-domain data, and was evaluated on the test set of the in-domain data in a 10-fold cross-validation manner.
Experimental results when the German part of the corpus was used as target and source of different language pairs in terms of precision, recall, and F-score
EN, English; DE, German; ES, Spanish; HU, Hungarian; ADAPT, language adaptation setting; CROSS, cross-language setting.
We also investigated what results could be achieved if the system was trained only on the added target sentences without using the source domain in the training process (ID). This model was also evaluated in a 10-fold cross-validation setting at the instance level.
Figure 1 outlines the process how we applied additional language data to identify each individual LVC in a running text.
Experimental results when the Spanish part of the corpus was used as target and source of different language pairs in terms of precision, recall, and F-score.
EN, English; DE, German; ES, Spanish; HU, Hungarian; ADAPT, language adaptation setting; CROSS, cross-language setting.
Experimental results when the Hungarian part of the corpus was used as target and source of different language pairs in terms of precision, recall, and F-score
EN, English; DE, German; ES, Spanish; HU, Hungarian; ADAPT, language adaptation setting; CROSS, cross-language setting.
6. Results
In this section, we will present the results of the methods applied on different languages in the 4FX corpus. We will also report the results of language adaptation and domain adaptation with additional language data.
6.1 Results of language adaptation
Tables 8–11 list the results for the language adaptation. Based on the in-domain 10-fold cross-validation results, our method can achieve relatively the same results on the Hungarian and English parts of the corpus with F-scores of about 65, and the same results on the German and Spanish parts of the corpus with F-scores of about 50. On each corpus, the language adaptation achieved an F-score that was higher than that of the dictionary lookup approach. Moreover, except for the Spanish case, the cross-experiments also yielded higher F-scores than in the dictionary lookup case.
Table 8 shows the results on the English corpus, where all the other three languages can contribute to the overall performance with the language adaptation. This method was most effective with an F-score of 66.30 with the highest precision score (81.97%), when we applied the union of the Spanish and German parts of the 4FX corpus as the source. In the case of the cross-experiments, the best results we got were when we trained our model on German. The difference between the average results of the CROSS and those of the dictionary lookup experiments was 7.99. The results of the baseline dictionary lookup method were noticeably exceeded by the ADAPT results.
We can see the results of German experiments in Table 9, where the largest difference is between the average language adaptation and cross-language experimental settings as the difference was 33.11 in terms of F-score. Here, our approach proved the most effective when we extended the German training set with the Spanish part of the 4FX corpus, since in this case, the F-score was 0.59 higher than the in-domain German results. The average difference between dictionary lookup and cross-experiments was 3.91, but with the best combination the gap was 15.34.
Table 10 lists the results for Spanish experiments. Here, our method was the most effective when the Spanish training set was extended with German. In this case, primarily the precision was affected by the language adaptation as the average precision score was 2.75 higher than the in-domain precision. Moreover, this difference in the precision score is 4.60 when the training set was extended with the union of the English and Hungarian corpora. The language adaptation results exceeded the cross-experiments by 19.05. But in this case, the dictionary lookup outperformed the cross-experiments as it achieved an F-score of 40.28.
Table 11 shows the results got on the Hungarian part of the 4FX corpus by using baseline dictionary lookup and our language adaptation. Language adaptation achieved the same F-score (65.25) with different precision and recall scores, when we extended the Hungarian training set with the English corpus and the union of the German and Spanish parts of the 4FX corpus. The difference between the average results of the language adaptation and the Hungarian in-domain experiments was 0.30, but the method generally worked better when the training set was not just extended with one language. The dictionary lookup approach got the highest precision, namely 85.86% but with the average cross-experimental setting we achieved an F-score that was 3.13 points higher.
The size of the target language data added to the source datasets greatly influenced the results as shown in Figure 2 with four typical settings in the four different languages. As we can see, language adaptation is effective when there is only a limited amount of target data available, since the target data contribute to the training process and the models produced increasing results.
Tables 12–15 show the usefulness of each individual feature type in four languages in our in-domain and cross-experiments. The performance scores of the features were compared with those obtained by applying all features described in Section 5.2. Here, we trained a J48 classifier with all of the features except that one, for each feature type. Then, we compared the performance to that got with all the features.
As our ablation analysis shows, each type of feature contributed to the overall performance in each language. We found only two exceptions. First, in English cross-experimental results, the statistical features had a detrimental effect and second, in the case of Spanish, the machine learning-based method could achieve a slightly better result (+0.03) without syntactic features.
From the tables for ablation analysis, we ascertain that the lexical features were the most important in all languages, as here it has the largest difference. For example, in English, the language adaptation ablation results obtained an F-score that was 39.2 points lower when the lexical features were ignored. However, these features also play an important role in German and Spanish but to a lesser extent. For instance, in the case of German we obtained an F-score that was 20 points lower when we omitted lexical-based features, but in this case, the orthographic and syntactic features had a bigger influence than they did for the other languages.
The effect of the size of the target language data on detecting LVCs in four different languages. English: the union of the German and Spanish parts of 4FX was the source. German: the Spanish part of 4FX was the source. Spanish: the German part of 4FX was the source. Hungarian: the English part of 4FX was the source.
In English, the precision scores were less affected by the cross-experimental ablation analysis compared to the in-domain experiments. In contrast, the omission of features had a greater effect on the precision scores in Hungarian cross-experiments than they did on the in-domain precision scores. Moreover, the language-specific features proved more useful in German and Spanish than they did in English and Hungarian.
6.2 Results of domain adaptation with additional language data
Table 16 lists the domain adaptation results got using additional language data on the English corpora, while Table 17 shows those for the Hungarian corpora. In the case of English, Wiki50 was the target, while the union of the English, German, and Spanish parts of the 4FX corpus was the source.
As Table 16 shows, when the gold annotated target language instances were increased in the source domain, the average difference between the in-domain and domain adaptation results consistently increased. When only 10% of target language was used in the source, the average difference in F-score was 0.09, and when all of the source data were used, it was 1.76. Moreover, in the latter case, we can outperform the in-domain results by 0.47. Here, the cross-domain experimental settings can achieve an F-score of 41.05, while the in-domain setting yielded an F-score of 53.89.
The usefulness of individual features in terms of precision, recall, and F-score using the English part of 4FX
The usefulness of individual features in terms of precision, recall, and F-score using the German part of 4FX
The usefulness of individual features in terms of precision, recall, and F-score using the Spanish part of 4FX
In the case of Hungarian, the short business news domain from the Szeged Corpus was applied as the target corpus, and the union of the English and Hungarian parts of the 4FX corpus was the source. The DAAL column of Table 17 lists the results obtained for the Hungarian domain adaptation with the additional language data task. Here, the average differences also consistently increased between in-domain results and domain adaptation with additional language data results when we extended the gold standard target language instances in the source. However, the most obvious difference was demonstrated when the whole of the source data was applied as the gold standard. Here, the average F-score difference is 2.72. Moreover, the in-domain setting achieved an F-score of 57.32, which was exceeded by the DAAL results of 1.83 more, while the cross-domain experiments yielded an F-score of 42.37.
The usefulness of individual features in terms of precision, recall and F-score using the Hungarian part of 4FX
Results of domain adaptation with additional language data on the English Wiki50 corpus in terms of precision, recall, and F-score
EN, English part of 4FX; ES+DE, the union of the Spanish and German parts of 4FX; Wiki50, Wiki50 corpus; ID, in-domain results; DAAL, domain adaptation with additional language data
The size of the target data added to the source data sets greatly influenced the results as shown using two settings in Figures 3 and 4. Figure 3 shows the Wiki50 target, cross-experiments and domain adaptation with additional language data results when the source was the whole set of union of the English, German, and Spanish parts of the 4FX corpus.
Results of domain adaptation with additional language data on the Hungarian short business news corpus in terms of precision, recall, and F-score
HU, Hungarian; EN, English; newsml, Hungarian short business news corpus; ID, in-domain results; DAAL, domain adaptation with additional language data.
The gap between the in-domain results and domain adaptation with additional language data results progressively decreases as the size of the dataset increases. But the gap between the results of CROSS and those of DAAL progressively increases with the amount of the data added.
Figure 4 lists the results by applying the Hungarian short business news domain as the target, cross-experiments and domain adaptation when the source was the union of the all of Hungarian and English part of the 4FX corpus. The results got from this model also exceeded the TARGET results, when all of the target instances were added to the training data as it yielded an F-score 1.83 that was higher than the in-domain results.
7. Discussion
Here we discuss our results achieved in different settings, namely, in-domain, cross-language, language adaptation and domain adaptation with additional language data settings.
7.1 In-domain experiments
Our machine learning method extensively outperformed the dictionary lookup baseline method in all the four languages in the 4FX corpus, which demonstrates that our syntax-based machine-learning method can be readily applied to automatic LVC detection in different languages.
Interestingly, our method can perform better in the case of English and Hungarian than German and Spanish. As Table 6 shows, the syntax-based candidate extraction method can extract only 60.80% of German LVCs, however, it extracted 87.10% of the Hungarian LVCs. In those languages, where the candidate extraction method proved to be less effective, the classification method reached lower recall scores and achieved modest F-score results. We think that the less effective dependency parser is probably responsible for the less impressive candidate extraction results on German and Spanish: the parser did not find any direct dependency relations in 29.01% of the German LVCs and 25.00% of Spanish LVCs. In contrast, in the case of English and Hungarian, the dependency parsers did not detect any syntactic relations for only 10% of LVCs. However, the automatic detection of German LVCs proved to be the most difficult task, as the dictionary lookup and the cross-language experiments also yielded an F-score of less than 20. It is mainly due to the relatively high ratio of the hapax LVCs, which is also a possible limit for the dictionary lookup method to achieve a higher recall score. Besides, the German light verbs were the most diverse, as we found 93 unique light verbs among the manually annotated German LVCs. This also prevents the machine learning-based method from providing good results as the lexical features are the most useful ones here.
The effect of the size of the target data on detecting LVCs in Wiki50. ADAPT, domain adaptation with additional language data setting; ID, training on a limited set of target data; CROSS, cross-domain setting; TARGET, in-domain setting.
The effect of the size of the target data on detecting LVCs in the short business news domain from Szeged Corpus. ADAPT, domain adaptation with additional language data setting; ID, training on a limited set of target data; CROSS, cross-domain setting; TARGET, in-domain setting.
Language similarity measure based on Kendall’ s coefficients
Another possible reason for the modest performance on German and Spanish is that the average occurrences of LVCs are fewer than in English and Hungarian. So, the machine learning-based method encounters fewer examples of the particular LVCs in these languages, which may hinder the learning process.
7.2 Cross-language and language adaptation experiments
The results of our cross-language experiments exceeded those of the dictionary lookup method. This highlights the fact that a machine learning-based model trained on a different language can be more effective than a target language dictionary lookup method. Moreover, to achieve a higher recall score, the dictionary lookup approach was primarily limited by the size of the available dictionaries, but this method still managed to yield a fairly good precision score. There was one notable exception: in the case of Spanish, the dictionary lookup method performed better than the cross-language experiments as it achieved a relatively high recall score. This may be attributed to the big size of the dictionary on the one hand and to the building principles behind the dictionary on the other, which were based on the theory of lexical functions (see, e.g., Melčuk Reference Mel’čuk1974), which was also taken into account when designing the annotation principles for 4FX.
Moreover, the cross-language training by itself did not prove sufficient in many cases. Therefore, the inclusion of annotated target language data into the training dataset was necessary to reduce the gap among different languages. As Figure 2 shows, by adding some target language data to the training dataset, it is possible to achieve similar results to those for in-domain settings.
The cross-experiments and language adaptation results showed that the proper selection of the source language largely influences the performance of the automatic detection of LVCs. The Kendall’s coefficients of the union of the light verbs were calculated to demonstrate the differences or similarities of languages, which is shown in Figure 5. This figure can help us to select the proper source language. Based on the Kendall’s coefficients, the German LVCs are the most similar to English, which accords with the cross-language experimental results in Table 8. Here, the model trained on the German part of the 4FX corpus achieved the best cross-language results with an F-score of 46.31.
With Figure 2, we would expect Hungarian cross-language experiment settings to perform the best on German since Hungarian proved to be the most similar based on the Kendall’s coefficient of the light verbs. But the results in Table 9 suggest that the English LVCs are the most similar to German, since the model trained on the English part of 4FX yielded higher cross-language results with an F-score of 24.12. Still, the precision scores of the German cross-language experiments demonstrate a clear relationship between Hungarian and German: the Hungarian setting achieved a much higher precision score with 85.37% than any of the other languages. So, some German examples are very similar to Hungarian LVCs—they are literal translations, such as in Verkehr bringen and forgalomba hoz (into traffic bring) “put into circulation” or in Kraft treten and hatályba lép (into force step) “enter into force”—and the model trained on the Hungarian examples was able to detect these with a relatively high precision. However, there were only a handful of such cases, which resulted in a relatively low recall score.
In the case of the Spanish cross-language experimental results, the Hungarian model proved to be the most successful. But, we were able to achieve the best results in the language adaptation experiments when German was selected as the source language, which was the most similar to Spanish according to the Kendall’s coefficients. In the Hungarian cross-language experiments, the two most similar languages, English and German yielded the best results.
As illustrated by our cross-language experiments, the model trained on the Hungarian part of 4FX mostly had a positive effect on the precision scores while the German model improved the recall scores. This may be due to the fact that German light verbs are the most diverse, hence, German data can provide many different examples of possible LVCs and also data sparsity for hapax LVCs in the target language may be resolved to little extent.
As Table 8 indicates, language adaptation outperformed the in-domain English results in every case. As the English part of the 4FX corpus contains the fewest gold annotated LVCs in the corpus, the machine learning-based method achieved a better result when we extended the training set with instances from other languages.
As regards German, the cross-language experiments performed the weakest, which is mainly due to the weak recall scores. This may be due to the facts that were mentioned in Section 7.1: the ratio of the hapax LVCs is relatively high in the German corpus and the light verbs were the most diverse here concerning the four languages examined. Based on the data in Table 5, the distribution of German LVCs is rather balanced and this language seemed to contain the most heterogeneous LVCs. Thereby, the machine learning-based model trained on the additional language model was not able to provide acceptable recall scores, as there were no LVCs with high frequency that may cover the majority of LVC occurrences. It is also seen that the cross-language experiment setting is the most successful when we trained our approach on the union of the English and Hungarian corpora. This is due to the specific characteristics of the two languages: when we applied the Hungarian cross-settings, our method yielded the highest precision score (85.37%), and achieved the best recall score with 25.04% when we trained it just on the English corpus.
It is well demonstrated how difficult the automatic detection of the Spanish LVCs is, as the best recall score was achieved in the in-domain setting. Here, the cross-language experiments only had a positive effect on the precision scores. This is probably due to the same fact as in German: namely, for Spanish the hapax LVCs and light verbs occurred in a relatively high ratio, so the additional language data could not assist the machine learning-based detection of rare examples. Also, the average precision score of the language adaptation results is just 2.75 points higher than in the in-domain case.
As for Hungarian, the most successful cross-language experimental results were achieved with English as the source language, and the union of the English and Spanish datasets had the most beneficial effect on the precision score. This may be explained by the fact that the English in-domain model could also achieve a very high precision score, so from the English dataset, the model may learn how to select genuine LVCs from the candidates. Still, applying German as the training set again had a beneficial effect on the recall score, most likely because of the heterogeneity of its LVCs.
7.3 Ablation analysis
As our ablation analysis revealed, the most important feature type was the lexical feature type. This is not surprising as most typical verbs in LVCs belong to a well-defined set, which are present in a high percentage of data in each language.
Syntactic features proved to be the most effective in the case of German. This may be attributed to the fact that the ratio of split LVCs is the highest in German, which means that components of LVCs are often not adjacent and syntactic information helps us a lot to match these parts.
In the case of Hungarian, we observe that for the language adaptation setting, recall improves with the omission of features, except for the lexical ones. This is again probably due to the fact that the English model mainly improves precision and its effect on recall is less clear.
As for language-specific features, it is noticeable that in the case of English, they improve precision but harm recall whereas they behave in the opposite manner in the case of Hungarian. This may be because in English, one of the language-specific features is whether the given verb is used as an auxiliary, which relies on POS-tags. Thus, selecting only those verbs that are not auxiliaries can improve precision because the set of candidate verbs is filtered. On the other hand, in Hungarian, the feature historical derivation yielded some false positive cases that included nouns of verbal origin but they are now used in a lexicalized meaning and do not form part of LVCs. These cases had a harmful effect on precision, however, this feature has a beneficial effect on recall as many nouns of verbal origin could be identified in this way.
7.4 Domain adaptation with additional language data
Our English and Hungarian cross-domain experimental results highlighted the fact that domain dependence is also important for the automatic detection of LVCs since the cross-domain results were always worse than the corresponding in-domain results. However, when there is only a limited amount of target data available, domain adaptation is more effective since the out-domain dataset also contributes to the training process. Moreover, training only on the amount of target data cannot achieve such outstanding results. On both languages, the domain adaptation experiments extended with additional language data exceeded the in-domain results when the machine learning-based model was trained on the entire set of gold standard source data with 100% of the target data. However, when we trained our model on the whole set of target domain, it did not seem to benefit from the silver standard out-domain data: it just confused learning since the in-domain results exceeded the domain adaptations. But, in other cases, the beneficial effect of the domain adaptation could be observed when we applied the silver standard as source domain.
8. Conclusions
In this paper, we presented a machine learning-based method that allows the full coverage identification of LVCs in raw texts of different languages. The method applied essentially depends on an English-based approach which was adapted to German, Spanish, and Hungarian with language-specific features. In order to see how much different languages affected each other, we also implemented a language-independent representation of LVCs and features.
Our method solved the problem in a two-step approach, so we implemented a standardized candidate extraction method on different languages in the first step. We found that the efficiency of the machine learning-based method mainly depends on the dependency parsers applied as the candidate extraction method was based on the dependency labels. The somewhat lower quality of parsing was responsible for our modest F-score results on German and Spanish, compared to those got for English and Hungarian, where we were able to achieve better results in terms of F-score. Furthermore, the automatic detection of German LVCs proved to be the most difficult task as the dictionary lookup, the cross-language experiments, and the in-domain machine learning-based method were the weakest here. This may be due to the facts that German light verbs were the most diverse, the ratio of the hapax LVCs is relatively high, the average occurrences of LVCs were fewer than these in English and Hungarian, and the ratio of hapax light verbs was much higher in German than other languages.
We also found that the proper selection of the source language largely influenced the performance of the automatic detection of LVCs in the target language. We saw that the similarity of the different languages based on the Kendall’s coefficient got by comparing the light verbs used in different languages can help us in our choice.
As our cross-language experimental results outperformed the dictionary lookup results, it highlights the fact that a machine learning-based model trained on data from (an)other language(s) can indeed achieve better results than a target language dictionary lookup method. However, the cross-language training by itself did not prove sufficient in many cases. This was why we applied a simple language adaptation technique to reduce the gap between different language pairs and we found that it was possible to achieve similar results as in the case of in-domain settings.
For English and Hungarian, we also experimented with cross-domain settings, which revealed that the domain dependence is also essential for the automatic detection of LVCs since the cross-domain results were always worse than the corresponding in-domain results. Nevertheless, when training on the combination of out-domain and additional language data, we were able to achieve practically the same results as in the case of in-domain training, which again demonstrates the portability of our model and usefulness of our standardized feature set across languages and domains. Thus, our approach can prove useful in cases where there is a small amount of annotated data for the given language or domain but there is an ample amount of annotated data for another language and domain. As for LVC detection, this means that if we would like to develop an LVC detector for another language, for example, Italian, we only have to annotate a few hundreds of Italian LVCs and apply language adaptation with one or more of the previously annotated corpora.
From a linguistic point of view, we argued that the very same feature representation of a given phenomenon might be suitable for typologically different languages as well. In this way, we were also able to make interesting comparisons across languages concerning the distribution of LVCs and light verbs in the 4FX corpus, which could assist theoretical research in contrastive linguistics and lexicology. The annotated corpus also allows one to study legal language in its entirety, or just LVCs in the legal domain.
In the future, we would like to improve our methods, with special regard to the candidate extraction phase for Spanish and German. we intend to test our approach on the PARSEME shared task corpora, which were prepared on the basis of the same annotation guidelines, hence provide a perfect testbed for cross-lingual comparisons. We also would like to extend our approach to other languages as well and we would also like to apply language adaptation techniques to other types of MWEs and even test our approach in other NLP tasks such as information extraction. We also think that our results may be incorporated in machine translation systems as well and improve their overall performance.
Acknowledgements
This research was supported by the UNKP-17-4 New National Excellence Program of the Ministry of Human Capacities.
