2.1 Cross-language information retrieval

Regarding translation resources for query translation, CLIR methods could be classified into four categories (Pirkola et al. Reference Pirkola, Hedlund, Keskustalo and Järvelin2001; Nie Reference Nie2010): methods that use either (1) bilingual dictionaries or (2) machine translators, or (3) comparable corpora, or (4) parallel corpora.

Among different methods for query translation that use parallel corpora, the methods that employ machine translation models are more popular and among different translation models, IBM model-1 (Brown et al. Reference Brown, Pietra, Pietra and Mercer1993) is more commonly used in CLIR (Nie Reference Nie2010). This model extracts p(f|e) from a parallel corpus. p(f|e) is the probability of translating a word e in the source language to a word f in the target language. Using this translation model, Nie et al. (Reference Nie, Isabelle, Plamondon and Foster1998) select the top N translation candidates for a given query and use these translations for building the query in the target language. This method first translates the query (translation step) and then uses the translated query for retrieving the related documents (retrieval step). The translation and retrieval steps can be also integrated within a unified framework (Berger and Lafferty Reference Berger and Lafferty1999). Lavrenko, Choquette, and Croft (Lavrenko et al. Reference Lavrenko, Choquette and Croft2002) use a parallel corpus for estimating the probability of relevance of each word in the target language with respect to a given query in the source language. Using these probabilities, they estimate the language model of a query in the target language.

Comparable-corpora-based methods use the association between words in aligned documents for estimating the relatedness of words in two different languages. The effectiveness of comparable-corpus-based methods for CLIR is investigated in two main studies which use comparable corpora for query translation (Tao and Zhai Reference Tao and Zhai2005; Talvensaari et al. Reference Talvensaari, Pirkola, Järvelin, Juhola and Laurikkala2008). Tao and Zhai (Reference Tao and Zhai2005) use comparable corpora for calculating an association score between words in two different languages. Talvensaari et al. (Reference Talvensaari, Pirkola, Järvelin, Juhola and Laurikkala2008) use the similarity score of documents in two different languages in addition to co-occurrence of words in the aligned documents to calculate the similarity score between the words in two different languages. In addition to these two most popular translation acquisition methods, there are other methods which try to extract and use translations achieved from comparable corpora in the language modeling framework (Rahimi and Shakery Reference Rahimi and Shakery2013; Shakery and Zhai Reference Shakery and Zhai2013).

Machine translators have also been exploited for translating queries and/or documents to the target language and finding related documents in the target language. Some of these research studies use machine translators for simply translating queries to the target language (Aljlayl and Frieder Reference Aljlayl and Frieder2001). The effectiveness of machine translation for Chinese–English CLIR has also been studied (Wu and He Reference Wu and He2012). The findings of this research show that translations derived from machine translation systems have high quality and accuracy in CLIR. Moreover, they utilize techniques such as relevance feedback methods to expand the translations achieved from a machine translator. This approach also has been exploited in other research studies such as Chen and Gey (Reference Chen and Gey2004). Jones and Lam-Adesina (2002) employ data fusion methods to merge the ranked lists achieved from two machine-translation-based CLIR systems.

2.2 Combining translation resources for CLIR

In order to achieve higher performance in CLIR, a number of studies have been carried out for employing multiple translation resources simultaneously. These methods could be categorized into two general categories: methods that assign weights to different resources and use a weighted combination of the resources for query translation (Lu, Chien, and Lee Reference Lu, Chien and Lee2004; Kadri and Nie Reference Kadri and Nie2007; Türe, Lin, and Oard Reference Türe, Lin and Oard2012; Azarbonyad, Shakery, and Faili Reference Azarbonyad, Shakery and Faili2013) and methods that use multiple resources for disambiguating translations of one another (Chen, Jiang, and Gey (Chen et al. Reference Chen, Jiang and Gey2000); McNamee and Mayfield Reference McNamee and Mayfield2002). The main issue with the methods of the first category is the cost of tuning the weights of different resources. The most simple weighting approach is fixing the weights of the resources based on some heuristics (Lu et al. Reference Lu, Chien and Lee2004). A more sophisticated approach (Kadri and Nie Reference Kadri and Nie2007) is learning the weights of different resources. The methods in the second category extract a number of translations from different resources and merge them to come up with a translation for the query in the target language.

2.3 Learning to rank for CLIR

Learning to rank (LTR) methods have been widely used in IR applications (Li Reference Li2011), and specifically in multilingual information retrieval (MLIR). Although MLIR and CLIR are very similar tasks, they have some differences. The main difference between MLIR and CLIR is that CLIR deals with the problem of retrieving documents in a language different from that of a query, but MLIR deals with the problem of retrieving documents in several languages different from the query language. Some studies use LTR in the merging step of MLIR (Tsai, Wang, and Chen Reference Tsai, Wang and Chen2008). After translating queries with a bilingual dictionary, they use an LTR method for merging retrieved document lists in each language and constructing a single list of documents. The main difference between this approach and our proposed method is that the former uses LTR to learn a merging method, not a ranking model. Gao et al. (Reference Gao, Niu, Zhou and Wong2009) employ machine learning methods for query translation. They extract features from translated query and document pairs and use these features for learning a ranking function. Effectiveness of using a multilingual parallel corpus for learning a scoring function in each language has also been studied by Usunier, Amini, and Goutte (Reference Usunier, Amini and Goutte2011). This approach uses a multilingual parallel corpus for predicting the paired labels of documents and then employs a bipartite ranking method for aggregating results as a ranked list. The main contribution of this method is using LTR for aggregating the bipartite rankings. However, our method directly uses LTR methods for learning a scoring function that can be used for ranking documents. Recently, Ghanbari and Shakery (Reference Ghanbari and Shakery2018) used LTR for training a ranking model for CLIR. They define various features for creating a model. Their main focus is on training local models for different query types which have similar characteristics, rather than creating a global model for all queries. Although, their defined feature set is similar to ours, they only use a parallel corpus for extracting cross-lingual features while we use different translation resources to perform CLIR. Moreover, our main focus (e.g., combining translation resources for CLIR) is different than theirs.

LTR has also been used in a few other cross-lingual settings such as cross-language question re-ranking and cross-language microblog retrieval. In an LTR-based cross-language re-ranking approach (Da San Martino et al. Reference Da San Martino, Romeo, Barroón-Cedeño, Joty, Maàrquez, Moschitti and Nakov2017), syntactic trees are constructed for both aligned source and target language questions and different features are extracted based on the similarity of syntactic trees of the questions and are used for ranking questions. Another study (Godavarthy and Fang Reference Godavarthy and Fang2016) tackles the task of cross-language microblog retrieval by mapping microblog messages in both source and target languages to a same semantic space and measuring the similarity of messages on the constructed space. Comparable corpora are also used for learning a cross-lingual word embedding (Vulić and Mones Reference Vulić and Moens2015). The learned word embeddings are used for constructing embedding of queries and documents. The created embeddings are then employed for estimating the similarity of queries and documents. Recently, online LTR has been also used for CLIR (Rahimi and Shakery Reference Rahimi and Shakery2017). Online LTR is different than offline LTR as the former is based on using implicit frequent user feedback to train a ranking model, while the latter creates a model based on the whole training set and does not update the model frequently.

3.1 Adapting monolingual LTR features for CLIR

One of the important features used in many LTR methods is frequency of query terms in documents. For estimating this feature in a cross-language setting, we should estimate the frequency of source language query terms in the target language documents. In this paper, Probabilistic Structured Queries (PSQ) approach (Darwish and Oard Reference Darwish and Oard2003) is used for mapping TFs from source language to target language. The PSQ approach uses the translation probabilities between source and target language terms to estimate the frequency of a source language term in a target language document. In this approach, the TF of word e in a source language S in document D_F in a target language F is calculated as:

$$ CLTF(e,{D_F}) = \sum\limits_{{f_i} \in {V_F}} p({\kern 1pt} {f_i}|e).TF({\kern 1pt} {f_i},{D_F}), $$(1)

where V_F is the target language vocabulary, f_i is a word in document D_F, p(f_i|e) is the translation probability estimated from translation resources, and T_F(f_i,D_F) is the frequency of term f_i in document D_F.

Another important feature used in LTR methods is IDF of query terms. Similarly, the PSQ approach could be used for estimating IDF of word e in the target document collection as follows:

$$ CLIDF(e) = \sum\limits_{{f_i} \in {V_F}} p({\kern 1pt} {f_i}|e).IDF({\kern 1pt} {f_i}), $$(2)

where IDF(f_i) is the inverse document frequency of term f_i. Many IR methods use TF and IDF when calculating similarity of the query and documents. Thus, using the features extracted using Equations (1) and (2), many retrieval methods such as Okapi BM25 (Robertson, Walker, and Beaulieu Reference Robertson, Walker and Beaulieu2000) or language modeling approach in the target language could be used for retrieving relevant documents.

Okapi BM25 score is one of the most frequently used features in LTR. For estimating the cross-lingual Okapi BM25 feature, we use the method proposed in Nie et al. (Reference Nie, Isabelle, Plamondon and Foster1998). Using the translation model, this method selects the top N translation candidates with the highest translation probabilities for the whole query, Q_S, which is:

$$ p({\kern 1pt} f|{Q_S}) = \sum\limits_{e \in {Q_S}} p({\kern 1pt} f|e)P(e|{Q_S}), $$(3)

where e is a query term and f is a target language word. It is further assumed that p(e|Q_S) is the same for every query word and therefore Equation (3) is simplified to:

$$ p({\kern 1pt} f|{Q_S}) \propto \sum\limits_{e \in {Q_S}} p({\kern 1pt} f|e). $$(4)

The language modeling score of documents given the query is another important feature in LTR methods. In this paper, the method proposed in Berger and Lafferty (Reference Berger and Lafferty1999) is used for calculating the language model score of documents for a given query. The score of a document D to a query Q could be estimated using cross-entropy as follows:

$$ Sc(Q,D) = \sum\limits_{t \in V} p(t|{\theta _Q})\log p(t|{\theta _D}), $$(5)

where θ_Q and θ_D are the language models of query Q and document D, respectively, and V is the vocabulary. The language models for Q and D are probability distributions over all terms in the collection given the terms in the query and document, respectively. The maximum likelihood estimation with a smoothing method is typically used for estimating these probability distributions. Using translation probabilities, the query language model is estimated as follows:

$$ p(t|{\theta _{{Q_S}}}) = \sum\limits_{s \in {V_S}} p(t|s,{\theta _{{Q_S}}})p(s|{\theta _{{Q_S}}}) \approx \sum\limits_{s \in {V_S}} p(t|s)P(s|{\theta _{{Q_S}}}), $$ (6)

where $ {\theta _{{Q_S}}}$ is the language model of query in the source language, V_s is the source language vocabulary, and p(t|s) could be estimated from translation resources. After estimating the query language model, the similarity of a document to the query could be determined as:

$$ Sc(Q,D) = \sum\limits_{t \in V} p(t|{\theta _Q})\log p(t|{\theta _D}) = \sum\limits_{t \in {V_F}} \sum\limits_{s \in {V_S}} p(t|s){p_{ML}}(s|{\theta _{{Q_S}}})\log p(t|{\theta _{{D_F}}}), $$(7)

where $ {p_{ML}}(s|{\theta _{{Q_S}}})$ is the maximum likelihood estimation of probability of a word s in Q_S. The document language model $ {\theta _{{D_F}}}$ in the target language should be smoothed by means of smoothing methods.

We adapt the standard features used in document retrieval (Nallapati Reference Nallapati2004) and also five other features which are common in LTR approach: document length, Okapi BM25 score, language model score with Jelinek–Mercer smoothing, absolute discounting smoothing, and Dirichlet prior smoothing. Using methods described in this section, we map these monolingual features to cross-lingual features. In Table 1, monolingual features and their mapped equivalents in CLIR are shown. In order to estimate the translation model, which is used in mapping the features, multiple translation resources are examined in Section 4.

Table 1.

Features used for learning the ranking function (In each column, the best result is bolded)

TF(q, D): frequency of term q in document D, |.|: size-of function, n: number of query terms, C: collection, C_F: target language collection, D_F: target language document, IDF(q): inverse document frequency of term q, and TF(q, C): frequency of term q in collection C.

3.2 Named-entity-based features

Named entities could be useful for estimating the similarity between queries and documents. The weights of different query terms are not equal in expressing the intent of the query. For example, if a query contains a general term and a specific name, usually the specific name is more informative for finding relevant documents. Named entities are usually very specific. Detecting named entities in queries and using them in CLIR can be very effective in improving the performance. Specific named entities may not be present in translation resources. In order to make use of named entities in CLIR, we detect and translate named entities by means of some external resources. We define four named-entity-based features and use them beside the mapped features described in Section 3.1 for learning a ranking model.

For finding named entities in queries, we use Stanford Named Entity Recognizer (Finkel, Grenager, and Manning Reference Finkel, Grenager and Manning2005) which is trained on the CoNLL data set with four entity types (Organization, Location, Person, and Misc). After recognizing each type of named entities in the source language, we use Google TranslatorFootnote ^a for translating them to the target language. We use the frequency of each named entity type, which is contained in the query, in documents as a feature. In order to prevent biasing this feature toward long documents, we normalize the frequency of named entities by the length of the documents:

$$ NETF(Q,D) = {{\sum\limits_{nm \in Q} F({T_{nm}},D)} \over {|D|}}, $$(8)

where nm is a named entity, T_nm is the translation of nm, F(T_nm,D) is the frequency of T_nm in document D, and |D| is the length of D. This feature is computed for the four kinds of named entities the Stanford NER toolkit can identify. Based on the four different entity types we extract four named-entity-based features. We estimate these features once for all query–document pairs and then use them in different LTR methods for CLIR.

Overall, we have 16 features: 12 of mapped features and 4 named-entity-based features. We estimate the 12 mapped features using different translation resources and use them for training the ranking models of LTR methods.

4.1 Parallel- and comparable-corpora-based CLIR

For performing CLIR based on parallel corpora, we employ Equation (4) which is described in Section 3.1 for query translation. We use the translated queries for calculating the BM25 score of documents. We employ IBM model-1 for estimating the translation model. Moreover, after estimating the translation probabilities between source and target language words, we use the translation probabilities for estimating the cross-lingual features.

We use the method proposed in Hashemi (Reference Hashemi2011) for extracting word associations from aligned documents in comparable corpora. This method uses similarity scores of aligned documents for extracting word associations. The output of this method is a dictionary that suggests a relatedness score for each pair of source–target language words. The basic intuition of this approach is that words co-occurring frequently in a lot of aligned documents are related to each other. The relatedness of words is captured using mutual information of words in aligned documents. This method extracts translation probabilities of target language words, p(f|e), with regard to all source language words. After extracting translations from comparable corpora using the described method, we use the translated query to implement BM25 IR method. Moreover, having the translation probabilities, it is straightforward to use them for mapping features and using them in an LTR setting for creating a ranking model.

4.2 Combining translation resources

Dictionaries, comparable corpora, parallel corpora, and machine translators have been widely used for query translation in CLIR. Each of these translation resources have some advantages compared to other resources. The main benefit of parallel corpora is their high accuracy for estimating the translation probabilities needed for query translation. Comparable corpora do not have the accuracy of parallel corpora, but they have query expansion effect in query translation process. Also, dictionaries have other benefits like their availability for many languages. The main issue with using dictionaries for query translation is their limited coverage. Machine translators are usually accurate in translating a complete sentence, but do not provide translation probabilities. Since each resource has its own advantages, combining translations extracted from different resources may lead to more accurate results.

Different approaches have been proposed for combining translation resources (Chen et al. Reference Chen, Jiang and Gey2000; McNamee and Mayfield Reference McNamee and Mayfield2002; Lu et al. Reference Lu, Chien and Lee2004; Kadri and Nie Reference Kadri and Nie2007) but most of them only use simple techniques for combining translation resources. One of the basic methods of resource combination is the linear combination method. This approach calculates the final translation probability of a source language word, e, to a target language word, f, as follows:

$$ p({\kern 1pt} f|e) = \sum\limits_{{R_i}} {\lambda _i}*{p_{{R_i}}}({\kern 1pt} f|e),\sum\limits_{{R_i}} {\lambda _i} = 1, $$(9)

where R_i is a translation resource, λ_i is the weight of the resource, and $ {p_{{R_i}}}({\kern 1pt} f|e)$ is the translation probability between e and f extracted from resource R_i. The probabilities are calculated using Equation (9), and then used for estimating the similarity of the query and documents.

Finding optimal weights of different resources is hard, especially when there are many translation resources available. In this case, simple methods for adjusting weights such as grid search are not efficient. To overcome this problem, we employ the LTR approach. We use features extracted from multiple resources together in order to construct the ranking model. In the ranking model, each feature gets a weight with regard to its importance in ranking the documents. The final score of a document regarding the query in this method is calculated as follows:

$$ Score({Q_s},{D_t}) = \sum\limits_{{R_i}} \sum\limits_{i = 1}^{\# {f_i}} {\lambda _{ij}}{f_{ij}}, $$(10)

where R_i is a translation resource, #f_i is the number of features extracted using R_i as the translation resource, f_ij is a feature, and λ_ij is the weight of f_ij. λ_ij’s are adjusted by means of LTR approaches. Using features extracted from multiple resources together, we can analyze the importance of each feature in ranking and by analyzing features we can determine the importance of each resource in the ranking. In this paper, a number of LTR methods such as AdaRank (Xu and Li Reference Xu and Li2007), IR-SVM (Cao et al. Reference Cao, Xu, Liu, Li, Huang and Hon2006), and Coordinate Ascent (Metzler and Croft Reference Metzler and Croft2007) are used for combining features.

5.1 Data sets

In this paper, Hamshahri collection is used for evaluating the proposed methods. 20M parallel corpus and University of Tehran Persian-English Comparable Corpus (UTPECC) comparable corpus (Hashemi, Shakery, and Faili Reference Hashemi, Shakery and Faili2010) are used for training the translation model of parallel-corpora-based methods and implementing comparable-corpora-based methods, respectively. Hamshahri collection consists of about 160,000 text documents in Persian and 100 queries in Persian and English. We use the title field of each English query. Table 2 shows the statistics of Hamshahri corpus. The 20M parallel corpus is constructed from four different parallel corpora: Roman parallel corpus (Mansouri and Faili Reference Mansouri and Faili2012), Iran Telecommunication Research Center parallel corpus (Jabbari et al. Reference Jabbari, Bakhshaei, Mohammadzadeh Ziabary and Khadivi2012), European Language Resources Association English–Persian parallel corpus (Mosavi Miangah Reference Mosavi Miangah2009), and a part of Mizan parallel corpus.Footnote ^b This corpus consists of 1,109,584 aligned sentences, which have about 20,000,000 words on each side. Table 3 shows the statistics of this corpus. UTPECC corpus is constructed from aligned news articles in Persian and English. In this corpus, BBC articles are aligned to Hamshahri articles. Table 4 shows the statistics of this corpus.

Table 2.

Hamshahri test collection statistics

Table 3.

20M parallel corpus statistics

Table 4.

UTPECC comparable corpus statistics

5.2 Evaluation metrics

In this section, we introduce evaluation metrics used in this paper for evaluating the proposed methods. We use Mean Average Precision (MAP), which is a common measure in IR, for evaluating our methods. Given k documents retrieved for query q, the Average Precision (AP) is calculated as follows:

$$ {\rm{AP}} = {{\sum\limits_{i = 1}^k Prec(i)*Rel(i)} \over {\rm{Number}\,\rm{of}\,\rm{relevant}\,\rm{documents}}}, $$(11)

where Rel(i) is an indicator function equal to one if the document at rank i is a relevant document, zero otherwise, and Prec(i) is precision at rank point i which is defined as:

$$ Prec(i) = {{RR(i)} \over i} $$(12)

where RR(i) is the number of relevant retrieved documents at rank point i. MAP is the mean of AP’s over all queries. Another measure that has been widely used in IR is P@n which is equal to Prec(n) described above. In this paper, we use P@5 and P@10 metrics for evaluating different methods.

5.3 Experimental results

In this work, the features described in Section 3 are extracted from judged documents for each query and are used for training LTR models. In addition to the judged documents, for each query, 10,000 unjudged documents are randomly selected from Hamshahri collection and the features extracted based on these documents are also used for constructing train and test sets. Unjudged documents are considered to be non-relevant to each query. Since the trained model will be tested on the whole collection containing judged and unjudged documents, unjudged documents should be considered in the training process. Since there are many more unjudged documents in the test set compared to the judged documents (relevant and non-relevant), the number of unjudged documents in training process is considered to be higher than the number of documents in other categories.

We use fivefold cross-validation for evaluating the proposed methods. The whole data set is partitioned into five folds. Each fold contains extracted features from 20 queries and their associated documents (judged documents and selected unjudged documents). We train LTR methods on four folds and use the fifth fold to evaluate the trained model.

We set up different experiments based on different translation resources. Also, we set up one monolingual experiment for evaluating the performance of LTR methods on monolingual IR. We have tested three different LTR methods [AdaRank (Xu and Li Reference Xu and Li2007), IR-SVM (Cao et al. Reference Cao, Xu, Liu, Li, Huang and Hon2006), and Coordinate Ascent (Metzler and Croft Reference Metzler and Croft2007)] for monolingual IR and CLIR methods. Our results indicated that all of LTR methods outperform traditional methods in both monolingual and cross-lingual IR cases. Among different LTR methods, AdaRank had the best performance and significantly outperformed other methods. Therefore, in this section, we only present the results of AdaRank method as the results of LTR methods.

5.3.1 Monolingual IR experiments

In these set of experiments, Persian queries are used for retrieving Persian documents. We implement BM25 and LTR methods using Persian queries. For training the models of LTR methods we use monolingual features listed in Table 1. In Table 5, the results of the first set of experiments are shown. As can be seen from this table, the LTR method outperforms Okapi BM25 for monolingual IR. Okapi BM25 is one of the features used for training the ranking model of AdaRank method. The performance of AdaRank in terms of MAP is 0.4531, which is 10% better than the performance of Okapi BM25.

Table 5.

Results of CLIR methods on Hamshahri collection (In each column, the best result is bolded)

Mono., Dic, PC, CC, and MT are monolingual, dictionary, parallel corpus, comparable corpus, and machine translator.

We have conducted statistical significance test (t-test) on the improvement of LTR method over the Okapi BM25. The results indicate that improvement of this method over Okapi BM25 in terms of MAP is statistically significant (p-value < 0.05).

5.3.2 Dictionary-based CLIR experiments

In these set of experiments, we use a dictionary for translating English queries to Persian. We implement Okapi BM25 model on the translated queries. Also, we implement the proposed feature mapping method using a dictionary and employ the mapped features alongside named-entity-based features for learning the ranking model of LTR methods. For translating English queries to Persian, we use Aryanpour English–Persian dictionary.Footnote ^c

The results of dictionary-based CLIR method are shown in Table 6. In this experiment, the queries are translated using the dictionary such that the first N translations for each English word are selected as its translation. As can be seen from Table 6, the best result of this method is achieved when N = 6. The translated queries are used for calculating the Okapi BM25 score of documents. The results of dictionary-based CLIR with N = 6 are selected as the baseline of dictionary-based method to be compared to other methods.

Table 6.

CLIR results on Hamshahri collection using dictionary (In each column, the best result is bolded)

%Mono shows the relative performance of the CLIR system compared to the performance of the monolingual system.

In order to increase the performance of dictionary-based CLIR, we use the dictionary for estimating the features expressed in Section 3. We consider the probability of translation of each English word e to its associated Persian words in the dictionary to be uniform, that is, $p({\kern 1pt} f|e) = {1 \over {{N_e}}}$, where N_e is the number of Persian words associated to the English word e. After estimating the features, we use them as training data for learning the ranking model of LTR methods. In Table 5, the results of this experiment in terms of MAP and P@n are shown. In this experiment, we considered N to be 6, which is the optimal value for N.

As the results show, LTR methods for dictionary-based CLIR outperform Okapi BM25 method. We have conducted statistical significance test (t-test) on the improvements of AdaRank method over Okapi BM25 method. The results show that the improvement of AdaRank over Okapi BM25 method is not significant in terms of MAP, P@5, and P@10. We believe that since the accuracy of dictionary for query translation is very low, combining different features estimated using the low accurate dictionary does not add any useful information to the Okapi BM25 feature.

5.3.3 Parallel-corpora-based CLIR experiments

In these set of experiments, we do CLIR based on parallel corpora. For each query word, we select its N most probable translation candidates and construct the query in the target language. We use IBM model-1 to extract a probabilistic dictionary from 20M parallel corpus. After translating queries, we implement Okapi BM25 method for calculating the similarity of queries and documents. Table 7 shows the results of these set of experiments for different values of N. As the results show, the best performance in terms of MAP and P@5 is achieved with N = 5. The performance of this method in terms of MAP is 0.302, which is 73% of the MAP of monolingual method.

Table 7.

CLIR results on Hamshahri collection using parallel corpus (In each column, the best result is bolded)

In order to improve the performance of parallel-corpora-based CLIR, we implement the method described in Section 4.1, which uses parallel corpora for extracting translations and employs the extracted translations for estimating the cross-lingual features. After estimating the features, we use the mapped features and named-entity-based features for constructing the ranking model of LTR methods. In Table 5, the results of this experiment are shown. The LTR method using parallel corpus outperforms using the parallel corpus for translating the queries and then performing monolingual retrieval. The performance of AdaRank in terms of MAP is 0.319, which is 6% better than using parallel corpus for CLIR. Conducted statistical significance test (t-test) shows that the improvements of AdaRank for CLIR over the basic method in terms of MAP are statistically significant (p-value < 0.05).

5.3.6 Multi-resource-based CLIR experiments

In these set of experiments, we implement the combination method described in Section 4.2 and study the effectiveness of different resources in CLIR. The results of different combinations of these resources are shown in Table 8. The performance of the LTR approach using a combination of resources is better than the performance of this method using each single resource. Moreover, the results show that the LTR-based combination method outperforms the linear combination method which is one of the most widely used combination methods.

Table 8.

LTR for combining parallel corpus (PC), comparable corpus (CC), machine translator (MT), and dictionary (Dic) results (the numbers in parentheses are %Mono) (In each column, the best result is bolded)

* The improvement achieved in that metric is statistically significant compared to the corresponding single-resource-based CLIR method.

Among different combinations, combination of parallel corpus, comparable corpus, and machine translator has the best performance. This shows that these resources are providing complementary signals for query translation. Parallel corpus and machine translator suggest the exact translations of the query and comparable corpus expands the query by related words and combining these translations improves the performance of CLIR. The results show that adding the features extracted using dictionary to the features extracted using parallel corpus does not have any effect on the performance of the LTR method. The features extracted using dictionary are very inaccurate as the translation probabilities are estimated naively in dictionary-based feature mapping. In contrast, the translation probabilities obtained from parallel corpus are more accurate. Therefore, adding low-quality features estimated from the dictionary cannot improve the ranking model constructed based on the features extracted from the parallel corpus. Moreover, the results show that adding dictionary-based features to the comparable-corpus-based features increases the performance of both dictionary and comparable corpus for CLIR. This result shows that unlike parallel-corpus-based LTR for CLIR, dictionary adds useful information to the ranking model of comparable-corpus-based LTR. Dictionaries are usually human made. Thus, by using the dictionaries, we can extract the exact translations of query words. Also, as the aligned sentences in parallel corpus are exact translations of each other, translations that are extracted from parallel corpus for queries are usually exact translations of query words. Therefore, dictionary and parallel corpus tend to translate queries accurately. However, dictionary has the shortcoming of ambiguity compared to parallel corpus, which results in its poor performance in CLIR. Unlike parallel corpus and dictionary, comparable corpus tends to expand the query and extract related words to the query words. Therefore, combining each of parallel corpus and dictionary with comparable corpus adds new information to the comparable corpus and improves the performance of comparable corpus in CLIR.

The results show that LTR-based combination approach outperforms the linear combination method. The grid search method is used for adjusting the weights of translation resources in the linear combination method. We have conducted statistical significance tests on the improvements of the LTR-based combination method over the linear combination method. Experiment results show that the improvements are statistically significant in all combinations.

As an additional baseline, we use the method proposed by Azarbonyad, Shakery, and Faili (Reference Azarbonyad, Shakery and Faili2014) which exploits LTR for translating queries from English to Persian and then performs CLIR. The main intuition of this approach is exploiting multiple translation resources for disambiguation, de-noising, and aggregating translations extracted from the resources. This method considers query translation as a ranking task in which given a word in the source language, the goal is to rank its translation candidates in the target language and select the top-N candidates as the translation of the source language word. To build the ranking model, they extract different features which reflect the relatedness of source and target language words. Then, using these features, an LTR method is employed to learn a translation model for CLIR. We use the same translation resources used for building our proposed combination model to create the translation model proposed in Azarbonyad et al. (2014). This method achieves MAP of 0.3217, P@5 of 0.562, and P@10 of 0.518 using all translation resources. The results indicate that our used LTR approach (PC+CC+MT in Table 8) outperforms this baseline in terms of all evaluation metrics. This indicates that it is more effective to first estimate query–document relevance signals using different translation resources and then combine them using an LTR approach. LTR for query translation (Azarbonyad et al. Reference Azarbonyad, Shakery and Faili2014) has a reasonable performance and it outperforms the linear combination method, but it cannot take full advantage of the different translation resources.