2.1 Cross-lingual embedding spaces

A vector space for words represents each word as a vector of a fixed dimensionality with the aim of grouping semantically similar words closer to each other in this space Rapp (Reference Rapp1995). Modern methods use neural networks for building such embedding spaces from raw text corpora (Bengio et al. Reference Bengio, Ducharme, Vincent and Jauvin2003). Out of many methods for building monolingual embedding spaces, this study primarily uses FastText (Bojanowski et al. Reference Bojanowski, Grave, Joulin and Mikolov2016), a recent approach, which combines the traditional skip-gram model with a model for building the embedding vectors for character n-grams within words. This incorporates some information from the subword level into the word embedding vector.

A commonly used model for building a cross-lingual embedding space is based on constructing a linear transformation matrix (TM) W for transforming one of the monolingual spaces to the other one by minimising the following objective:

(1) $$\mathop {\min }\limits_{\bf W} \sum {||{\bf W}{e_i} - {f_i}|{|^2}}$$

where e _i ∈ E and f _i ∈ F are the respective embedding vectors in the two languages for words, which are supposed to be translations of each other according to the seed dictionary. This study uses a method for building W via SVD (Artetxe, Labaka, and Agirre Reference Artetxe, Labaka and Agirre2016), which ensures that W is an orthogonal matrix built using a closed-form solution:

(2) $${\bf W} = {\bf V} \times {{\bf U}^T}$$

when V and U are the matrices from the SVD factorisation of F × E^T; see Artetxe et al. (Reference Artetxe, Labaka and Agirre2016) for justification and discussion.

2.2 Cross-lingual mapping using cognates

The method for cross-lingual mapping across related languages in this study consists of three steps:

1. (1) automated collection of seed bilingual dictionaries;

2. (2) determining weights for the Levenshtein distance (LD) from the seed dictionaries;

3. (3) alignment of monolingual embeddings by linear transformation using orthogonalisation and WLD.

The seed dictionaries can be provided by word alignment of large parallel corpora. In a low resource setting, the seed dictionaries can be obtained from the titles of interlinked Wikipedia articles in two languages (iWiki links)^{Footnote a}; see examples of aligned titles in Table 2. This helps in modelling scenarios when few parallel texts are available, such as for the Polish–Russian pair. Even though Polish is included in Europarl, and Russian is in the UN corpus, very few reliable resources are available for the Polish–Russian pair itself. The titles have been word-aligned using FastAlign (Dyer, Chahuneau, and Smith Reference Dyer, Chahuneau and Smith2013). The resulting word-level dictionaries have been filtered against the respective frequency lists, since the Wikipedia titles are dominated by relatively infrequent proper names, which are not representative of the properties of the general lexicon. Table 2 lists a random selection of the word forms aligned for the Polish–Russian pair.

Table 2.

Alignments from Wikipedia for titles and words

In addition to providing the training lexicon, a seed dictionary can also be used to provide a character-level model for matching the cognates via WLD; see the part of Table 2 for examples of character alignment. The pairs of words from the training dictionary have been aligned on the character level (again using FastAlign in this study) to produce the probabilities of regular correspondences between the characters in the two languages. The character alignment model is particularly important for establishing orthographic similarity when the two languages use different character sets, such as the case for Polish and Russian. For example, the characters with the highest probability for translating the Russian characters ф and л into Polish are, respectively, f and ł.

In the end, the standard edit operations for computing the traditional normalised LD can be weighted by the probabilities of their character-level alignments:

(3) $$WLD({s_e},{s_f}) = {{\sum {_{(e,f) \in al({s_e},{s_f})}} \,(1 - p(f|e))} \over {\max \,(len({s_e}),\,len({s_f}))}}$$

where s _e and s _f are words in the two languages, al is a set of their alignments and p(f|e) is the probability from the character alignment model. The distance is normalised by the length of the longest word.

Given that even correctly aligned words from the Wikipedia titles for related languages are not necessarily cognates, for example, wskaźnik versus индекс (‘index’) from Table 2, the process of getting the Levenshtein weights ran in two steps. In the first step, an initial estimate of the character translation probabilities was produced from all word pairs in the seed dictionary. This was used for assessing the rough WLD between them. The most likely cognates according to this rough WLD were used as the input for the second iteration of character-level alignments. The WLD threshold for choosing the most likely cognates was determined for each language pair individually. Repeated application of these steps did not result in any improvements in detecting cognates.

The value of either LD or WLD can be used as a factor for scoring the translation suggestions:

(4) $$score({s_e},{s_f}) = \alpha \,cos\,({v_e},{v_f}) + \,(1 - \alpha )(1 - WLD({s_e},{s_f}))$$

where s _e and s _f are the words in the two languages, v _e and v _f are their embedding vectors in the cross-lingual embedding space, while α is the relative weight of the cosine similarity. Unlike the cosine similarity measure, the WLD value is greater for more remote strings.

While the combined score is useful for producing bilingual dictionaries, it does not affect the bilingual embedding space by itself. A closed-form solution for orthogonalisation as used in (2) helps in improving alignment quality in the general case. However, it does not allow adjusting the TM by taking into account the orthographic similarity between the cognates. An easy way for incorporating this information into the cross-lingual embedding space is by aligning the entire lexicons from the cross-lingual space using the WLD score from (4) and selecting the most similar words in this list. This far longer lexicon can be used instead of the seed dictionary for producing a new TM from (2) for re-alignment of the already aligned cross-lingual space from the previous step. The rationale for this iteration is that we want to minimise the distance between the known cognates while preserving the orthogonality of the TM. Again, while repeated application of these steps is possible, it did not produce better results, so the experiments below present the results obtained after two iterations.

2.3 Experimental set-up

This paper reports two sets of experiments. One experiment involved a replicable setting for the English–Italian language pair with the standardised embeddings and training/test dictionaries initially developed for Dinu, Lazaridou, and Baroni (Reference Dinu, Lazaridou and Baroni2014) and used in Artetxe et al. (Reference Artetxe, Labaka and Agirre2016). Even though English and Italian are not closely related languages (English is a Germanic language, but Italian is from the Romance family), a large number of English words are borrowings from Romance languages, primarily from French and Latin, so the WLD approach could work for the En–It pair as well. The test dictionary from Dinu et al. (Reference Dinu, Lazaridou and Baroni2014) includes both cognate word pairs, such as academy/accademia, and non-cognate pairs, such as absolve/esimere or abysmally/malo, which are also often questionable translation equivalents. Therefore, a cognate-only version of the En–It test set was produced by retaining only the words with the WLD value above 0.5, reducing the En–It test dictionary from 1869 down to 818 entries.

In addition to the standardised embeddings as used in Dinu et al. (Reference Dinu, Lazaridou and Baroni2014), Artetxe et al. (Reference Artetxe, Labaka and Agirre2016), a new set of embeddings produced by FastText has been added to the English–Italian experiments (labelled as FT in Table 3). The FT embeddings have been the basis for the experiments with the Slavonic languages.

Table 3.

Prec@1 for En–It dictionary induction

The experiments with the Slavonic languages also emphasise the low-resource setting, when large parallel corpora for the seed dictionaries are not always available, so the seed dictionaries for building the transformation matrices and the WLD weights came from the iWiki links (the Italian seed dictionary used in Dinu et al. (Reference Dinu, Lazaridou and Baroni2014) and Artetxe et al. (Reference Artetxe, Labaka and Agirre2016) was derived from aligning Europarl).

2.4 Experimental results

The results listed in Table 3 confirm that orthogonalisation (Artetxe et al. Reference Artetxe, Labaka and Agirre2016) and global correction (GC; Dinu et al. Reference Dinu, Lazaridou and Baroni2014) improve the accuracy of translation detection in comparison to the baseline of Mikolov, Le, and Sutskever (Reference Mikolov, Le and Sutskever2013), the figures reported in bold are significant at p < 0.005. Embedding vectors produced by incorporating subword information (marked by FT in Table 3) also make a considerable positive impact. Adding the constraint of having orthographic cognates (LD or WLD) improves the accuracy of dictionary induction further, often by a substantial margin. Even for the English–Italian pair, where the languages operate over the same alphabet, WLD outperforms LD because it assigns a very low cost to more common substitutions, for example, x → s or j → g (examined → esaminato or Jerusalem → Gerusalemme).

The best value of α, the relative weight to balance the contribution between the cosine similarity and the WLD, was estimated at 0.73 using a development set which was randomly extracted from the training dictionary. The same value of α = 0.73 has been used throughout the remaining experiments. Relying exclusively on the orthographic similarity (α = 0) leads to relatively poor results.

Given that the FT+Orth+WLD combination results in consistently better performance, the results of dictionary induction across Slavonic languages are shown only for this set-up (Table 4). The row labelled #Cognates lists the number of WLD cognates retrieved for the second iteration of alignment. The amount of useable cognates depends on the size of the Wiki corpora used for training, see Table 1, as well as on the typological distance between the languages. Comparison of the Slavonic dictionary induction results to the English–Italian pair shows even more significant improvements through the use of WLD, occasionally from 0.429 to 0.840 for the Slovenian–Croatian pair. The Wikipedia corpus used for Croatian is quite small for reliable training of monolingual embeddings, so incorporating the WLD score contributes to improving the initial deficiencies of its space.

Table 4.

Dictionary induction results for Slavonic languages

The FastText vectors of 300 dimensions built from the Wikipedias for the selected Balto-Slavonic languages (Belarussian, Czech, Croatian, Lithuanian, Polish, Slovak, Slovene, Ukrainian) have been transformed into a shared Panslavonic embedding space. For convenience of running cross-lingual experiments, English has also been added to the shared embedding space. In spite of the fact that it is not a closely related language, its alignment to the Slavonic languages benefits from the WLD because of a large number of cognates such as the names of locations, personal names and borrowings. Another shared embedding space was produced for selected Romance languages.

2.5 True cognates and false friends

It is well known that even closely related languages have a number of false friends, for example, Mist in German means ‘manure’ unlike mist as used in English. However, a closer look at the list of cognates shows that there is a cline of cases:

1. (1) consistently false friends, for example, bezcenny means ‘worthless’ in Polish and ‘invaluable’ in Czech;

2. (2) partial false friends, for example, žena can mean either ‘wife’ or ‘woman’ in a number of Slavonic languages, for example, Croatian, while its cognate жена in Russian always means ‘wife’;

3. (3) actual cognates with uncommon divergent senses, for example, similarly to жена in Russian, in Polish żona means ‘wife’, while rarely it can also mean ‘woman’.

Therefore, the boundary between true cognates and false friends is quite flexible. This can lead to some disagreement between the annotators with respect to what constitutes false friends; see also a discussion in Fišer and Ljubešić (Reference Fišer and Ljubešić2013).

Monolingual word embeddings are built to reflect the similarity of the most common contexts via the distance between the embedding vectors, so the false friends are likely to have fairly distant vectors, as indicated by low cosine similarity values. However, the WLD reflects the similarity of the word forms, thus leading to the possibility of selecting false friends as possible translation equivalents. Therefore, the Panslavonic embedding space has been tested against available lists of Slavonic false friends to determine the amount of non-cognate noise introduced through the use of WLD.

A useful test bed is provided by the False Friends of the Slavist,^{Footnote b} which covers most of the language pairs for the Panslavonic set, even though its coverage differs across the language pairs. The first two columns in Table 5 list the false friends for the Russian–Czech direction provided in the dictionary. They can be ranked by their similarity scores (Column ‘Cos’) with the top words corresponding to consistently false friends, as their contexts typically differ. The words at the bottom of the list tend to be actual cognates, which have been included in the gold standard lists because they also have some divergent uses. While the words at the bottom of the lists can be potential false friends, corpus evidence for the most common senses suggests that their divergent semantic components are uncommon, at least in the Wikipedia corpus used for building the embeddings. Often there is a mismatch between dictionary definitions and the actual corpus use. For example, while the Russian word запах ‘smell’ is neutral in its dictionary sense, the majority of its collocations are negative (‘unpleasant’, ‘pungent’, ‘foul’, similarly to the collocations of the word odour in English), thus leading to its embedding vector being closer to zápach in Czech, which means ‘unpleasant smell’.

Table 5.

Ordering false friends in cognate lists

Higher orthographic similarity (lower WLD) increases the final score for all false friends. However, the consistently false friends have very low cosine similarity scores, so that the weighted sum needs to compete with other vectors, which are closer semantically. The last four columns in Table 5 list the Czech vectors closest to the Russian keywords according to the plain cosine measure, as well as its weighted sum with the WLD ( α = 0.73). The correct dictionary translations are indicated in bold, while the partially correct translations, such as запах versus zápach are in italics. In some cases, the WLD helps in correcting the raw cosine measures (seven instances, e.g., slovenský vs. slovinský), while in three cases using the weighted sum deselects the correct choice, but only when the initial similarity score was high (prst ‘finger’ vs. palec ‘thumb’). We can conclude that the WLD score tends to be helpful even in the difficult case of dealing with false friends.

3.1 Morphology prediction

3.1.1 Prediction of syncretism

The previous section shows that a procedure for aligning the cross-lingual embedding spaces can benefit from taking the similarity between the languages into account. So far the proposed procedure assumed a one-to-one mapping, namely that one form in the donor language corresponds to one form in the recipient language. While the problem with homonymy and polysemy of translation equivalents is important in the general case, this problem is relatively minor in related languages, because their words tend to keep the same distribution of meanings with relatively few exceptions; see the study of false friends above.

However, a far more common problem concerns differences in syncretism, that is, when one form can serve several grammatical functions. For example, the verbs in French have the same endings for the first- and third-person forms, while these forms are different in Spanish:

Fr:je/il anticipe versus Es:yo anticipo/el anticipa

Therefore, a single form in French needs to be similar to two forms in Spanish.

Syncretism is very common in Slavonic languages as well. Table 6 shows the different case endings for the Russian and Ukrainian adjectives. In Russian, all feminine non-nominative forms of зеленый (‘green’) have the same ending, while the endings in Ukrainian differ in each case. The reverse is true for the masculine dative and locative forms, which are different in Russian and identical in Ukrainian. So cross-lingual mapping between the forms needs to address the problem of variations in syncretism even across closely related languages.

Table 6.

Forms of adjectives in Russian and Ukrainian

3.1.2 Experimental set-up

A possible way of addressing this problem is by inferring information about morphology from the embeddings. It is known that the embeddings do keep information about the underlying morphology of the word forms, for example, Belinkov et al. (Reference Belinkov, Durrani, Dalvi, Sajjad and Glass2017). Therefore, we can set the task of predicting morphological properties from the embeddings. For example, we train a model to predict the case, gender and number for the two fairly close embedding vectors from the Panslavonic space:

ru зелёному=(−0.047−0.032−0.101 0.007 0.021−0.046 0.0066 0.095…)

→ Case=Dat|Gender=Masc,Neut|Number=Sing

uk зеленому=(−0.044−0.062−0.137−0.035−0.019 0.058 0.106 0.017…)

→ Case=Dat|Gender=Masc,Neut|Number=Sing

→ Case=Loc|Gender=Masc,Neut|Number=Sing

This expertimental setting helps in two ways. First, it tests the possibility to determine morphological properties within each language even after the cross-lingual transformation in order to assess the difference between the forms or to assign the right translation given a context. Second, it can help in populating the lexicons for POS taggers and parsers. Training corpora, especially for lesser-resourced languages, are quite small, see the corpus sizes in Table 1, while the prediction set-up using embeddings benefits from more contexts available in large raw text corpora.

Table 7 demonstrates the difference between the training and development parts of the UD corpora with respect to their lexicon. The smaller corpora have a substantial rate of out-of-vocabulary (OOV) words, which makes the tagging task harder, especially given that their tagging models are based on very sparse data.

Table 7.

Proportion of OOV words in the lexicons

The experimental set-up tested in this study involves predicting properties for nouns, adjectives and verbs from the Panslavonic vectors (300 dimensions) using the UD training sets for training and their development sets for testing morphological predictions. The UD test sets have been reserved for testing the accuracy of POS tagging and parsing. Prediction has been done by a multilayer perceptron with a single hidden layer of 150 neurons using tanh as the activation function and the Adam optimiser. Experiments with other hyperparameter settings did not change the results significantly. Two models have been tested:

R training using the original UD lexicon for each recipient language;

D training using cross-lingual embedding by transfer from related donor languages:

Cs→Pl,Sk, Ru→Be,Sk,Uk

Given the multi-label set-up, the evaluation metric is Average Precision for prediction (Sorower Reference Sorower2010). For example, when the model predicts four labels for a word form, three of which are correct, the precision for this prediction is 0.75.

3.1.3 Prediction results

Table 8 presents the results of prediction. In this table and in the discussion in the following, R stands for Recipient, D for Donor, D, O for the Donor part corresponds to predictions using the joint embedding space produced via orthogonal transform as in Artetxe et al. (Reference Artetxe, Labaka and Agirre2016) and D,W corresponds to the join embedding space produced by using WLD-induced cognates.

Table 8.

Morphology prediction results

Column #T_R indicates the number of tags in the training corpus for an individual recipient language, while #T_D indicates the number of tags in the donor language corpus. If the recipient corpus is small, for example, for Be, it covers only a small portion of possible tags. The number of examples available for training can be significantly increased via the donor language; see Columns Train_R and Train_D (many more examples are available in the donor corpora, so addition of examples was limited to provide at most 400 examples per tag). The donor language also provides more examples per individual tag; see Columns PerT_R and PerT_D. The test examples (Test) were selected from the development parts of the respective UD corpora for words not attested in the training corpora.

Table 8 shows that prediction usually improves by taking more data from the donor language. When the initial training set is very small, as it was the case for Belarussian, the improvement is dramatic, for example, from 3% to 68% for Belarussian verbs. The original Belarussian UD corpus contains merely 13 examples per verbal tag on average, which is not enough for training a classifier. Comparison of the AP_D,O versus AP_D,W columns (Orthogonalisation vs. WLD) shows overall improvement.

In the case of Czech, the UD tags make heavy use of features specific to the Czech training corpus, for example, Style (with such values as Colloquial, Archaic, Rare, etc) and NameType (Geo, Given name, Surname, etc), which are not used in the available feature sets in other related languages. These two specific morphological attributes have been removed before training. However, the number of Czech tags is still quite high; compare the numbers for #T_R versus #T_D in Table 8 for Polish and Slovak (#T_D is for Czech as the Donor). In the end, many Czech tags do not contribute to predicting the tags for Polish and Slovak in the cross-lingual setting. Another observation is that the gold standard is derived from an annotated corpus, which does not necessarily cover the entire paradigm for each test item. This means that the prediction model often produces correct results without receiving credit for this. For example, антропологический (‘anthropological’) in Russian in the gold standard corpus is annotated as

ADJ Case=Nom|Gender=Masc|Number=Sing

while the predicted annotation is equally correct:

ADJ Animacy=Inan|Case=Acc|Gender=Masc|Number=Sing

3.2 Named-entity recognition

3.2.1 Training set-up

The cross-lingual embedding space has been also tested through the NER task, which is aimed at detecting and labelling all occurrences of person names, organisations or locations. This is a convenient downstream task for which there are existing methods and test sets. Recently, various neural network approaches produced very convincing results for NER (Collobert et al. Reference Collobert, Weston, Bottou, Karlen, Kavukcuoglu and Kuksa2011). A particular implementation used in the extrinsic evaluation experiment reported in the following is based on a sequence tagging method, which combines bidirectional LSTM with CRF for making the final prediction (Lample et al. Reference Lample, Ballesteros, Subramanian, Kawakami and Dyer2016). Each word is represented by its embedding vector from the shared embedding space, in addition to other easily available features, such as character-level embeddings or the presence of capitalisation. The taggers for individual languages were trained from an existing NER-annotated corpus from Krek et al. (Reference Krek, Erjavec, Dobrovoljc, Holz, Ledinek and Može2012) in Slovenian using the Panslavonic embedding space.

Small samples from each language have been added to the Slovenian training corpus in order to provide at least some information for the character-level embeddings. The small additional samples were derived from the Wikipedia title names in the respective languages for the articles which categories matched such patterns as ‘Births’ (for person names), ‘Organizations’ (for organisations) and ‘Countries’ or ‘Villages’ (for locations, since the Wikipedia articles usually lack a more generic category of locations). For example, an entry for a sample of Russian person names looks like:

The entry contains the likely first and following elements of a named entity (B-Per and I-Per, respectively), and it ends with a third person verb, which helps in learning typical conditions when a named entity ends. The most common verbs and prepositions were used as the ending elements as selected from the respective UD corpora.

3.2.2 BSNLP NER shared task

The NER shared task at BSNLP’17 contained two separate test sets with no training sets for individual languages. One test set was based on news reporting about the European Commission, and another one on newswires concerning Donald Trump. The baseline system (Piskorski et al. Reference Piskorski, Pivovarova, Šnajder, Steinberger and Yangarber2017) was based on large gazetteers developed by the JRC, while the only other submission covering all Slavonic languages from JHU (Mayfield, McNamee, and Costello Reference Mayfield, McNamee and Costello2017) was based on projection of NER labels via word-aligned parallel corpora, see Table 9, as well as a brief explanation of the projection approach in Section 4.

Table 9.

F1 strict prediction scores for NER at BSNLP

The shared embedding space is surprisingly efficient, the figures reported in bold are significant at p < 0.005. The training corpus was for Slovenian, so it provides the upper baseline for language adaptation. Czech, Croatian and Polish are sufficiently similar typologically, so the accuracy on those languages is only slightly below what has been achieved for Slovenian. Russian and Ukrainian are East Slavonic languages, further away typologically from the rest, which is probably the main reason for the markedly lower accuracy of transfer from the Slovenian training set. Across all languages, the NER tagger has a problem with detecting relatively long NERs, which are common in the EC test set, such as The European Convention for the Protection of Human Rights and Fundamental Freedoms, while the accuracy is higher on general newswire texts. Overall, the results are considerably lower than what has been achieved for English, which can be explained by much richer morphology of the Slavonic languages, as well as by a relatively small training set. Despite such limitations, the transfer model which only used the Slovenian training corpus was on average more successful than the projection-based model.

3.3 Genre classification

Text classification is one of the commonly used tasks in NLP. A more specific task concerns classification of texts into genres (Santini, Mehler, and Sharoff Reference Santini, Mehler, Sharoff, Mehler, Sharoff and Santini2010), since genre annotation provides useful information for understanding kinds of texts a corpus consists of in addition to understanding the structure of its topics via Topic Modelling (Blei, Ng, and Jordan Reference Blei, Ng and Jordan2003; Sharoff Reference Sharoff, Sharoff, Rapp, Zweigenbaum and Fung2013). Unlike topic modelling, which usually uses unsupervised topic discovery via detection of keywords, the relationship between topics and genres is not well defined, since keywords from the same topic are often used in texts of different genres. Instead, genre classification requires a supervised approach to learn the association between stylistic features and genre labels.

A supervised approach needs a training set, which might be available for some specific languages and specific genre classification schemes, but not for others. The Language Adaptation framework can be used to solve this problem as well: training is done using the available donor resources within the shared cross-lingual embedding space, while the resulting model is applicable to the recipient language. As an example of such study, a Russian genre-annotated corpus (Sharoff Reference Sharoff2018) has been applied to classify Ukrainian texts into genres.

For evaluating the resulting classifier, a small testing corpus is still required. The Ukrainian corpus for this study has been collected from the Web to provide a sample of the major genres represented in the respective Russian corpus; see Table 10. For the ease of interpretation, the category labels given in brackets in Table 10 roughly correspond to the categories of the Brown Corpus (Kučera and Francis Reference Kučera and Francis1967), whenever possible. The Personal category (missing in the Brown corpus) primarily contains personal blog entries and personal messages from social networks.

Table 10.

Genre-annotated corpora

In comparison to other supervised text classification set-ups, such as sentiment analysis, genre classification can be biased by the topical words in the training corpus (Petrenz and Webber Reference Petrenz and Webber2010). A convenient representation, which can use cross-lingual embeddings and at the same time can have the capacity to generalise a genre across topics represented in the training set, is a mixed feature set (Baroni and Bernardini Reference Baroni and Bernardini2006), which is produced by replacing the less frequent words with their POS codes, while leaving the most common words in their original form. The POS codes have been taken from the UD set to ensure their transfer across languages. For example, a review text (in English for illustration purposes)

converts into a mixed representation as

This representation makes it easier to compare this review snippet to other reviews without relying too much on the specific keywords and numerical values, while it keeps important lexical features for detecting genres; see Petrenz and Webber (Reference Petrenz and Webber2010) for further discussion concerning the importance of non-topical representations for genre classification.

As for the machine learning approach, the genre classification experiment reported here uses a simple feedforward network inspired by FastText Joulin et al. (Reference Joulin, Grave, Bojanowski and Mikolov2017). In this set-up, we start with pre-trained word embeddings from Section 2.2 to build a document embedding representation, doc2vec. Then, simple feedforward neural networks are used for multi-labelled text classification. This method has been shown to be robust and efficient in a number of sentiment classification tasks for English, while achieving comparable accuracy in comparison to more complicated neural models based on CNN or LSTM (Joulin et al. Reference Joulin, Grave, Bojanowski and Mikolov2017). The specific implementation in this study is based on Keras.^{Footnote c}

Table 11 presents the results of classification in terms of average precision, which was the objective for optimising the training pipeline. Given the vast amount of texts on the Web, optimising for precision helps in extraction of useful sample texts in a specific genre, in contrast to retrieving all texts in this genre. In the multi-label context, the overall quality of classification can be described in terms of its Hamming loss, which computes the proportion of irrelevant predictions Sorower (Reference Sorower2010), thus the lower the better.

Table 11.

Genre classification results: Precision for Russian and Ukrainian

The first two columns in Table 11 (marked as CV) show the results of training classifiers on the respective training corpora with 10-fold cross-validation. The bigger Russian corpus quite predictably produces a much better model. The last two columns show the results of training on the Russian corpus with the two versions of the cross-lingual embedding space with and without WLD. In the same pattern as with the NER task, transferring data from the donor language usually helps, and the transfer accompanied with the WLD cognates helps even more. For example, fiction, legal and instructive texts can be detected reliably, so the genre classifier is useful for selecting their samples from the Ukrainian Web. At the same time, the resulting Ukrainian models suffer from the mismatch between the original Russian training set and the Ukrainian testing sample. In particular, the academic texts in the Ukrainian testing corpus came primarily from popular science sources, while the Russian model has been trained on a range of research articles.