1 Introduction
Noun compounds are short phrases comprising two or more nouns such as ‘apple pie’ and ‘information gathering.’ Noun compounds are very common and ubiquitous in everyday language and technical documents. Baldwin and Tanaka (Reference Baldwin and Tanaka2004) reported that about 3 per cent of all word tokens are used as part of noun compounds in English and Japanese corpora. Research on noun compounds is important in any discipline relevant to language. In particular, computer understanding of noun compounds is important for many natural language processing (henceforth NLP) applications, such as question answering, machine translation, and information retrieval (Butnariu et al. Reference Butnariu, Kim, Nakov, Ó Séaghdha, Szpakowicz and Veale2010). For example, in order to improve the retrieval performance for the query ‘heart disease,’ an IR system would benefit from a semantic interpretation engine that determines what words are semantically related to heart disease. Recently, a number of computational studies have been made on the interpretation of noun compounds (e.g., Girju et al. Reference Girju, Moldovan, Tatu and Antohe2005; Kim and Baldwin Reference Kim, Baldwin, Dale, Wong, Su and Kwong2005; Costello, Veale and Dunne Reference Costello, Veale and Dunne2006; Kim and Baldwin Reference Kim and Baldwin2007; Butnariu and Veale Reference Butnariu and Veale2008; Nakov Reference Nakov2008; Tratz and Hovy Reference Tratz and Hovy2010).
According to computational lexical semantics Jurafsky and Martin (Reference Jurafsky and Martin2008), at least two approaches are involved in computing with word meanings: i.e., word sense disambiguation and word similarity computation. In the case of noun compounds, they are specified as follows:
• Compound disambiguation determines which sense of constituent words is used, and identifies the semantic relation holding between (the senses of) constituent words in a noun compound. For example, we must identify the semantic relation holding between the words (i.e., apple and pie) of the noun compound ‘apple pie.’
• Similarity computation computes the semantic relatedness or similarity between a noun compound and other words (or compounds), which is used for identifying taxonomic (e.g., synonym and hyponym) and associative relations. For example, we compute the degree of semantic relatedness between a noun compound ‘apple pie’ and the word ‘tart.’
Compound disambiguation has been a main topic of research in existing studies on noun compounds, while similarity computation has received only marginal attention. Some studies (e.g., Kim and Baldwin Reference Kim, Baldwin, Dale, Wong, Su and Kwong2005; Ó Séaghdha and Copestake Reference Ó Séaghdha and Copestake2007; Nakov and Hearst Reference Nakov and Hearst2008; Ó Séaghdha and Copestake Reference Ó Séaghdha and Copestake2009) have employed similarity computation to identify the semantic relation between the constituent words in a noun compound. However, the similarity computed in these studies differs from the similarity expected by the similarity computation involved in computational lexical semantics. This difference of similarity is clearly described by Turney (Reference Turney2006). According to Turney (Reference Turney2006), there are at least two kinds of similarities: attributional similarity Footnote 1 and relational similarity. Attributional similarity is the correspondence between attributes and its degree is computed between two words. In the case of complex nominals including noun compounds, attributional similarity is computed by treating complex nominals as single words. Relational similarity is the correspondence between relations and its degree is computed between two word pairs. For example, the compound ‘chocolate milk’ has a higher attributional similarity to the compound ‘morning coffee’ than to the compound ‘plastic bag,’ because both milk and coffee are beverages and thus share many attributes. On the other hand, ‘chocolate milk’ has a higher relational similarity to the compound ‘plastic bag’ than to the compound ‘morning coffee,’ because the relation between chocolate and milk and the relation between plastic and bag are highly similar or identical, i.e., N 1 is a material of N 2. Existing studies on noun compounds have addressed a method of computing relational similarity to identify semantic relations, but have paid little attention to attributional similarity.
The problem of lack of interest in attributional similarity becomes more serious when we consider a particularly intriguing aspect of noun compounds that they can yield emergent properties or emergent meanings. In the psychological literature (e.g., Hampton Reference Hampton1987; Murphy Reference Murphy1988; Wilkenfeld and Ward Reference Wilkenfeld and Ward2001), emergent properties are defined as those that are salient in interpreting a noun compound, but not salient in representing its constituent words (i.e., either the head noun or the modifier). For example, people often think that ‘pet birds’ have the property talks, but they do not think that pets and birds by themselves have this property. Hence, the property talks is an emergent property of ‘pet birds.’ On the other hand, ‘pet birds’ also have the property sings, but this property is not emergent because it is also a salient feature of the head noun ‘birds.’ Similarly, the sense grid of the compound ‘power system’ is an emergent meaning because grid is not likely to be listed as a characteristic of either the head ‘system’ or the modifier ‘power.’ Emergent properties or meanings have been examined extensively and observed repeatedly in the psychological studies of conceptual combinations (i.e., noun compounding) Hampton (Reference Hampton1987); Murphy (Reference Murphy1988); Coulson (Reference Coulson2001); Wilkenfeld and Ward (Reference Wilkenfeld and Ward2001) and metaphors Becker (Reference Becker1997); Utsumi (Reference Utsumi2005). These findings indicate that noun compounding is not a fully compositional process and involves some sort of non-compositionality. Such ‘semi-compositional’ meanings cannot be computed solely by the compositional process of compound disambiguation; they should be explained within the process of attributional similarity computation. Thus, this paper aims at proposing and evaluating a method of attributional similarity computation for noun compounds. (In the rest of this paper, we use the generic term ‘similarity’ to specifically refer to attributional similarity.)
For the purpose of similarity computation, this paper employs a semantic space model (Bullinaria and Levy Reference Bullinaria and Levy2007; Landauer et al. Reference Landauer, McNamara, Dennis and Kintsch2007; Padó and Lapata Reference Padó and Lapata2007; Turney and Pantel Reference Turney and Pantel2010), in which each word is represented by a high-dimensional vector and the degree of semantic relatedness between any two words can be easily computed, for example, as the cosine of the angle formed by their vectors. Semantic space models are computationally efficient in representing meanings of words, because they take much less time and less effort to construct meaning representation, and they can provide a more fine-grained similarity measure between words compared to other representation methods such as thesauri (e.g., WordNet). Semantic space models are also plausible as a psychological (or cognitive) model of mental lexicon or semantic memory; a number of studies have shown that the vector-based representation achieves remarkably good performance for simulating human verbal behavior, such as similarity judgment and semantic priming (Bullinaria and Levy Reference Bullinaria and Levy2007; Landauer et al. Reference Landauer, McNamara, Dennis and Kintsch2007). Hence, they are advantageous for computing the similarity of emergent meanings of noun compounds.
Therefore, the basic question to be answered is how a proper vector representation of a noun compound should be computed in a semantic space model. One possible and simple way of doing this is to treat noun compounds as individual words; vectors for noun compounds are constructed directly from the corpus, just as vectors for words are constructed. This method is expected to compute a proper semantic representation of noun compounds, but suffers from one serious limitation: it cannot deal with unfamiliar noun compounds that do not occur in the corpus. This drawback is all the more serious considering the corpus-based finding that many noun compounds occur only a few times (approximately 45–60 per cent occur only once) Baldwin and Tanaka (Reference Baldwin and Tanaka2004) and the empirical finding that people easily comprehend unfamiliar compounds (e.g., Wisniewski Reference Wisniewski1996; Gagné Reference Gagné2000).
An alternative way of computing compound vectors is to combine word vectors for constituent words (i.e., the head noun and modifier) of a noun compound. Several algorithms (i.e., centroid, predication, comparison, and dilation) have been devised for vector composition Kintsch (Reference Kintsch2001); Mitchell and Lapata (Reference Mitchell and Lapata2008, Reference Mitchell and Lapata2010); Utsumi (Reference Utsumi2011), but their semantic validity has not been examined in a systematic way. Hence, this paper examines the applicability of these algorithms to noun compounds, by computing the similarity of noun compounds in English and Japanese and comparing their performances.
The rest of this paper is organized as follows. Section 2 reviews some previous studies relevant to this paper, focusing on noun compound interpretation and compositional semantics of semantic spaces. Section 3 explains two semantic space models used as a computational framework for computing the similarity of noun compounds. In addition, several algorithms for vector composition are introduced, which in Section 4 are compared in terms of the quality of noun compound vectors. Section 4 presents the method and results of the computer experiment conducted for evaluating the ability of the vector composition algorithms to compute the attributional similarity of noun compounds. In the experiment, three tasks (and measures) are used for evaluation: ranking of words semantically related to noun compounds, correlation between human similarity judgment and similarity computation, and semantic classification of compound-word pairs. Section 5 discusses some implications of the simulation results, as well as some issues for future studies. Finally, Section 6 provides concluding remarks for this study.
2 Related work
2.1 Noun compound interpretation
As mentioned in Section 1, noun compound interpretation involves two approaches, i.e., compound disambiguation and similarity computation.
Compound disambiguation refers to the task of determining the semantic relation holding between constituent words of a noun compound. For example, at the very least, possible semantic relations for the noun compound ‘headache pill’ are cause (i.e., ‘pill that causes headaches’), and purpose (prevent) (i.e., ‘pill that prevents headaches’). An algorithm that interprets noun compounds must determine possible relations for a given noun compound, and also rank these relations in order of plausibility.
Several previous studies (e.g., Rosario and Hearst Reference Rosario and Hearst2001; Girju et al. Reference Girju, Moldovan, Tatu and Antohe2005; Kim and Baldwin Reference Kim and Baldwin2006, Reference Kim and Baldwin2007) addressed the problem of compound disambiguation using a predefined set of semantic relations, and consequently they treated it as a multi-class classification problem, for which a number of efficient methods are available. For example, Nastase and Szpakowicz (Reference Nastase and Szpakowicz2003) proposed a hierarchical set of 30 relations grouped into five categories, while Girju et al. (Reference Girju, Moldovan, Tatu and Antohe2005) defined a flat list of 35 general semantic relations, only 21 of which turned out to be found in noun compounds. Recently, Tratz and Hovy (Reference Tratz and Hovy2010) presented a more fine-grained taxonomy of 43 semantic relations. Task 4 of SemEval-2007 (i.e., the fourth international workshop on semantic evaluations), aiming at testing the methods for classifying semantic relations between nominals (including noun compounds), includes seven relations (Girju et al. Reference Girju, Nakov, Nastase, Szpakowicz, Turney and Yuret2007, Reference Girju, Nakov, Nastase, Szpakowicz, Turney and Yuret2009). A wide variety of computational methods have been proposed for identifying appropriate semantic relations for a given noun compound: semantic scattering (Moldovan et al. Reference Moldovan, Badulescu, Tatu, Antohe and Girju2004; Girju et al. Reference Girju, Beamer, Rozovskaya, Fister and Bhat2010), word sense disambiguation Kim and Baldwin (Reference Kim and Baldwin2007), the Web-based n-gram model Lapata and Keller (Reference Lapata and Keller2005), support vector machines Girju et al. (Reference Girju, Moldovan, Tatu and Antohe2005), a maximum entropy classifier Tratz and Hovy (Reference Tratz and Hovy2010), associating sense pairs and semantic relations Rosario and Hearst (Reference Rosario and Hearst2001); Costello et al. (Reference Costello, Veale and Dunne2006), verb semantics Kim and Baldwin (Reference Kim and Baldwin2006), among others.
An alternative approach to compound disambiguation regards compound disambiguation as a process of paraphrasing according to which noun compounds are paraphrased by combinations of verbs and prepositions Kim and Baldwin (Reference Kim and Baldwin2006); Nakov and Hearst (Reference Nakov and Hearst2006); Butnariu and Veale (Reference Butnariu and Veale2008); Nakov (Reference Nakov2008); Butnariu et al. (Reference Butnariu, Kim, Nakov, Ó Séaghdha, Szpakowicz and Veale2010); Kim and Nakov (Reference Kim and Nakov2011). For example, the noun compounds ‘apple pie’ and ‘apple store’ can be paraphrased as ‘a pie that is made of apples’ and ‘a store that sells Apple products.’ This approach does not require a predefined set of semantic relations, which are sometimes criticized for being arbitrary and not able to provide a complete list. In addition, Nakov (Reference Nakov2008) and Butnariu and Veale (Reference Butnariu and Veale2008) have demonstrated that Web mining methods are effective in collecting paraphrasing information. Due to these advantages, the paraphrase-based approach has recently become a dominant approach in the field of noun compound interpretation. This is reflected by the fact that noun compound paraphrasing was included as part of a set of tasks run in the recent workshop on semantic evaluation, SemEval-2010 Butnariu et al. (Reference Butnariu, Kim, Nakov, Ó Séaghdha, Szpakowicz and Veale2010).
Concerning similarity computation, some studies on compound disambiguation have employed relational similarity to determine whether two noun compounds are based on the same semantic relation (e.g., Kim and Baldwin Reference Kim, Baldwin, Dale, Wong, Su and Kwong2005; Ó Séaghdha and Copestake Reference Ó Séaghdha and Copestake2007; Nakov and Hearst Reference Nakov and Hearst2008; Ó Séaghdha and Copestake Reference Ó Séaghdha and Copestake2009). A number of methods for computing relational similarity have been devised: for example, the use of WordNet similarity between constituent words of noun compounds Kim and Baldwin (Reference Kim, Baldwin, Dale, Wong, Su and Kwong2005) or between verbs that express semantic relations Kim and Baldwin (Reference Kim and Baldwin2006), Dice coefficients of Web-based feature vectors based on context verbs and prepositions of noun compounds Nakov and Hearst (Reference Nakov and Hearst2008), and a hybrid method that combines attributional similarity between constituent words and relational similarity between noun compounds Ó Séaghdha and Copestake (Reference Ó Séaghdha and Copestake2007, Reference Ó Séaghdha and Copestake2009); Nakov and Kozareva (Reference Nakov and Kozareva2011).
However, these studies have not addressed a method for computing attributional similarity for noun compounds,Footnote 2 and thus, to the best of the author’s knowledge, there are no studies investigating how attributional similarity of noun compounds can be computed. The lack of research on this topic may be due to the seemingly appropriate assumption that the computation of the attributional similarity of noun compounds does not require techniques or methods specific to noun compounds; a general method of similarity computation (i.e., word similarity) can be applied without modification to noun compounds by treating them as single words. However, as mentioned in Section 1, this assumption is incorrect, because noun compounding is a highly productive process, and thus an infinite number of unfamiliar compounds can be generated (e.g., Clark, Gelman and Lane Reference Clark, Gelman and Lane1985; Baldwin and Tanaka Reference Baldwin and Tanaka2004; Ó Séaghdha and Copestake Reference Ó Séaghdha and Copestake2007). Furthermore, noun compounds often have semi-compositional meanings, which may not be possible to derive from a simple combination of meanings of the constituent nouns. Therefore, automatic interpretation of noun compounds requires a method for computing the attributional similarity of noun compounds, or complex nominals in general, that can deal with the emergence of meanings. This is what this study aims to accomplish.
Note that most of the previous studies on noun compounds (including this study) deal only with two-noun compounds (i.e., noun compounds comprising two nouns), but some noun compounds include longer noun sequences that comprise more than two nouns (e.g., ‘information retrieval system’ and ‘world health organization’). Interpretation of these longer noun compounds requires the additional process of structural disambiguation, which is referred to as bracketingLauer (Reference Lauer1994, Reference Lauer1995); Girju et al. (Reference Girju, Moldovan, Tatu and Antohe2005). This process determines which part of the compound is the head and which the modifier. For example, the compound ‘information retrieval system’ is left-bracketed as ‘(information retrieval) system,’ which implies that ‘system’ is the head and ‘information retrieval’ is the modifier. Conversely, ‘world health organization’ is right-bracketed as ‘world (health organization),’ which implies that ‘health organization’ is the head and ‘world’ is the modifier. The bracketing problem is tackled in some studies Lauer (Reference Lauer1994, Reference Lauer1995); Girju et al. (Reference Girju, Moldovan, Tatu and Antohe2005); Lapata and Keller (Reference Lapata and Keller2005) by a corpus-based model. Lauer (Reference Lauer1994, Reference Lauer1995) demonstrated that corpus statistics for word adjacency and dependency are effective in bracketing noun compounds. Nakov and Hearst (Reference Nakov and Hearst2005) improved the corpus-based n-gram method by using search engine statistics, and Bergsma et al. (Reference Bergsma, Pitler and Lin2010) examined the advantages of a Web-scale n-gram model.
2.2 Semantic interpretation using semantic spaces
In order to compute the similarity for larger linguistic units (e.g., typically phrases, sentences, or paragraphs) than those represented in semantic spaces (e.g., typically words), it is necessary to devise a method for generating a vector representation of a larger unit from its constituent words.
The standard method for vector composition in semantic space models is to compute the centroid of the constituent word vectors. The centroid algorithm has been shown to yield very useful results in a number of applications of latent semantic analysis (e.g., Landauer and Dumais Reference Landauer and Dumais1997; Foltz, Kintsch and Landauer Reference Foltz, Kintsch and Landauer1998). However, it has the serious drawback that word order (and thus, a semantic role) is completely ignored, and thus the algorithm wrongly computes the same vector for a noun compound and its reversed compound (e.g., ‘apartment dog’ and ‘dog apartment’).
Kintsch (Reference Kintsch2000, Reference Kintsch2001) proposed a predication algorithm for computing intuitively plausible and contextually dependent vectors of the proposition (i.e., the sentence) with the predicate argument structure. Kintsch demonstrated that the predication algorithm is effective for metaphor comprehension, semantic disambiguation, and other applications. Utsumi (Reference Utsumi2011) also proposed a comparison algorithm for computing the meaning of metaphors as a model of Gentner’s Reference Gentner1983 comparison process comprising structural alignment and inference projection. However, it is not clear whether the predication and comparison algorithms are also effective in interpreting noun compounds, especially in processing the emergence of meanings.
Mitchell and Lapata (Reference Mitchell and Lapata2010) compared several composition algorithms with respect to the correlation between two similarity judgments for the same pairs of phrases, namely human judgment and vector-based judgment. The phrases used in their study included noun compounds such as ‘development plan’ and ‘oil industry.’ Thus, their study can be regarded as an application of semantic space models to noun compounds, and their findings can be compared with the results of this study for the sake of discussion. However, their study was not intended to propose a method for noun compound interpretation, and thus, they did not directly examine the meanings of noun compounds including emergent meanings. They only employed the similarity between noun compounds (e.g., ‘oil industry’ and ‘computer company’) for evaluating composition algorithms. In addition, they did not test the comparison algorithm, which may be effective in interpreting noun compounds.
Although not addressing noun compounds specifically, a number of studies on multiword expressions (i.e., short phrases comprising two or more words such as ‘kick the bucket,’ which subsume noun compounds) have employed a semantic-space-based similarity computation to identify non-compositional multiword expressions (Schone and Jurafsky Reference Schone and Jurafsky2001; Baldwin et al. Reference Baldwin, Bannard, Tanaka and Widdows2003; Katz and Giesbrecht Reference Katz and Giesbrecht2006; Giesbrecht Reference Giesbrecht, Rudolph, Dau and Kuznetsov2009; Guevara Reference Guevara2011; Reddy, McCarthy and Manandhar Reference Reddy, McCarthy and Manandhar2011). However, these studies assume that a correct vector for multiword expressions is constructed directly from the corpus by treating multiword expressions as individual words. This assumption implies that these studies cannot deal with unfamiliar compounds that do not occur in the corpus, which as mentioned in Section 1, is a serious problem when processing noun compounds. In addition, some of these studies also use vector composition – e.g., simple addition Schone and Jurafsky (Reference Schone and Jurafsky2001); Katz and Giesbrecht (Reference Katz and Giesbrecht2006); Guevara (Reference Guevara2011); Reddy et al. (Reference Reddy, McCarthy and Manandhar2011), multiplication Giesbrecht (Reference Giesbrecht, Rudolph, Dau and Kuznetsov2009); Guevara (Reference Guevara2011); Reddy et al. (Reference Reddy, McCarthy and Manandhar2011), and convolution Giesbrecht (Reference Giesbrecht, Rudolph, Dau and Kuznetsov2009); Guevara (Reference Guevara2011) – to compute the vector for multiword expressions, but they do not regard vector composition as a method for generating a plausible vector representation for multiword expressions. They simply employed the combined vectors to measure decomposability (i.e., the degree of semantic similarity between a multiword expression and its constituent words). Hence, they did not examine (or intend to examine) the plausibility of the combined vectors as the meaning representation of multiword expressions.
3 Computational model
3.1 Semantic space model
The semantic space model is based on two main assumptions. One assumption is that the meaning of each word wi can be represented by a high-dimensional vector u(wi) = (u i1, u i2, …, uin), that is, a word vector. These n real-valued components define the lexical meaning of the word. The second assumption is that the degree of semantic relatedness sim(wi, wj) between any two words wi and wj can be computed using a similarity function of their word vectors. This study uses as the similarity function the cosine of the angle formed by their vectors u(wi) and u(wj), which is the most widely used in semantic space models.
Semantic spaces (or word vectors) are constructed from large bodies of text (i.e., corpus) by observing distributional statistics of word occurrence. The method for constructing semantic spaces generally comprises the following three steps.
(1) Initial matrix construction: nw content words in a given corpus are represented as r-dimensional initial vectors whose elements are frequencies (i.e., the number of times a certain event relevant to the words occurs), and an nw by r matrix A is constructed using nw word vectors as rows.
(2) Weighting: The elements of the matrix A are weighted.
(3) Smoothing: The dimension of the row vectors of A is reduced from the initial dimension r to n.
As a result, an n-dimensional semantic space including nw words is generated. The last step of smoothing is optional: When the matrix is not smoothed, n = r (i.e., the initial dimension r becomes equal to the dimension n of the semantic space).
Numerous methods have been proposed for these steps of constructing semantic spaces (for an overview, see Padó and Lapata, Reference Padó and Lapata2007; Turney and Pantel, Reference Turney and Pantel2010). Among them, latent semantic analysis (henceforth LSA; Landauer and Dumais, Reference Landauer and Dumais1997; Landauer et al., Reference Landauer, McNamara, Dennis and Kintsch2007) is relatively popular, and is also adopted in this study to construct semantic spaces. LSA uses the frequency of words in a document to compute an initial matrix (Step 1), whose dimension r is equal to the number of documents. For Step 2, LSA often uses the tf-idf family of weighting schemes. In the tf-idf family, the weight is calculated by the product of the local weight based on the term frequency and the global weight based on the inverse document frequency. Although many functions have been proposed for calculating local and global weights (for an empirical comparison of different weighting functions, see Nakov et al., Reference Nakov, Popova and Mateev2001), in this paper we use the following function, i.e., the product of the logarithm of the word frequency and the entropy Quesada (Reference Quesada, Landauer, McNamara, Dennis and Kintsch2007):
\begin{eqnarray}
u_{ij} &=& \log (tf_{ij}+1) * \left( 1 + \frac{\sum _{k=1}^{r}P_{ik}\log {P_{ik}}}{\log {r}} \right)
\end{eqnarray}
\begin{eqnarray}
P_{ij} &=& \frac{tf_{ij}}{\sum _{k=1}^r tf_{ik}}
\end{eqnarray}
Another family of methods, which relies on word co-occurrence frequency, has also been widely used for constructing semantic spaces Burgess and Lund (Reference Burgess and Lund1997); Schütze (Reference Schütze1998); Bullinaria and Levy (Reference Bullinaria and Levy2007); Recchia and Jones (Reference Recchia and Jones2009). The most important difference from LSA is that a word–word matrix is constructed as an initial matrix at Step 1, whose elements are frequencies of word co-occurrences within a ‘window’ spanning any number of words. If we assume that the context is given by the set of words in the window, a word–word matrix can be regarded as a word–context matrix. The weighting scheme for a word–word matrix that is widely used and generally achieves good performance is pointwise mutual information (henceforth PMI; Church and Hanks, Reference Church and Hanks1990; Bullinaria and Levy, Reference Bullinaria and Levy2007), which is defined as
\begin{eqnarray}
pmi(w_i,w_j) &=& \log _2 \frac{p(w_i,w_j)}{p(w_i)p(w_j)}
\end{eqnarray}
\begin{eqnarray}
p(w_i,w_j) &=& \frac{cf_{ij}}{\sum _{i=1}^{n_w}\sum _{j=1}^{n_c}cf_{ij}}
\end{eqnarray}
\begin{eqnarray}
p(w_i) &=& \frac{\sum _{j=1}^{n_c}cf_{ij}}{\sum _{i=1}^{n_w}\sum _{j=1}^{n_c}cf_{ij}}
\end{eqnarray}
\begin{eqnarray}
p(w_j) &=& \frac{\sum _{i=1}^{n_w}cf_{ij}}{\sum _{i=1}^{n_w}\sum _{j=1}^{n_c}cf_{ij}}
\end{eqnarray}
\begin{equation}
u_{ij} = \left\lbrace \begin{array}{@{}l@{\quad }l@{}}pmi(w_i,w_j) & \text{(if $pmi(w_i,w_j)>0$)} \\
0 & (\text{otherwise}). \end{array}\right.
\end{equation}
In summary, we use two methods for constructing semantic spaces, namely, LSA and PPMI. The reason for comparing two kinds of semantic spaces in this paper is that we would like to examine whether and how vector composition algorithms depend on the characteristics of semantic spaces.
3.2 Computing the vector for noun compounds: Algorithms for vector composition
The vector for a noun compound can be computed by using vector composition algorithms. For a noun compound C comprising two nouns, i.e., the head noun H and the modifier noun M, the vector p = (p 1, …, pn) for the noun compound C is computed by
\begin{equation}
{\bf $p$} = f({\bf $h$},{\bf $m$}),
\end{equation}
In this study, as well as in some other studies (e.g., Baldwin and Tanaka Reference Baldwin and Tanaka2004; Kim and Baldwin Reference Kim, Baldwin, Dale, Wong, Su and Kwong2005), the right noun of a compound (e.g., pie of ‘apple pie’) is referred to as the head noun, and the left noun (e.g., apple of ‘apple pie’) is referred to as the modifier noun. This definition of head is based on the syntactic concept of headedness (i.e., the syntactic head of a compound is almost always the final constituent of that compound). In most cases, it coincides with the semantic definition of headedness that the head is the word that determines the semantic category of the compound.Footnote 3
In the rest of this section, six composition algorithms that are compared in an evaluation experiment are presented with an example of computation using the following hypothetical vectors.
\begin{equation}
{\bf $h$} = (0,\, 10,\, 4,\, 1,\, 6), \hspace*{14.22636pt} {\bf $m$} = (1,\, 8,\, 4,\, 4,\, 0).
\end{equation}
3.2.1 Centroid
The simplest but most widely used method for vector composition in semantic space models is to compute the centroid of constituent word vectors by
\begin{eqnarray}
{\bf $p$} &=& \frac{{\bf $h$} + {\bf $m$}}{2}.
\end{eqnarray}
3.2.2 Multiplication
An alternative and simpler method for vector composition is to use the multiplicative function (11), in which each component of one vector is multiplied by the corresponding component of another vector Mitchell and Lapata (Reference Mitchell and Lapata2010):
\begin{eqnarray}
{\bf $p$} &=& {\bf $h$} \odot {\bf $m$}
\end{eqnarray}
\begin{eqnarray}
p_i &=& h_i \cdot m_i.
\end{eqnarray}
Mitchell and Lapata (Reference Mitchell and Lapata2010) demonstrated that the multiplication algorithm better predicts human similarity judgment between two-word phrases. Hence, this study tests whether their result is due to the ability of the multiplication algorithm to compute a vector for a noun compound of high semantic quality, or it is an artifact of the specific task or semantic spaces used.
3.2.3 Circular convolution
Circular convolution is a function that compresses the tensor product of two vectors, which is defined as
\begin{eqnarray}
{\bf $p$} &=& {\bf $h$} \circledast {\bf $m$}
\end{eqnarray}
\begin{eqnarray}
p_i &=& \sum _{j=0}^{n-1} h_{j\bmod {n}} \,\cdot \, m_{(i-j)\bmod {n}}.
\end{eqnarray}
Circular convolution is often used as a method of generating semantic representations of individual words that contain syntactic or word-order information Plate (Reference Plate2003); Jones and Mewhort (Reference Jones and Mewhort2007). In this study, similar to Mitchell and Lapata (Reference Mitchell and Lapata2010), we regard circular convolution as an alternative method of vector composition.
Also note that the three algorithms (i.e., centroid, multiplication, and circular convolution) presented so far are insensitive to word order (or syntactic structure), which is a serious drawback of vector composition methods. On the other hand, the three composition algorithms explained below can generate different vectors depending on the order of the constituent words.
3.2.4 Predication
The predication algorithm was proposed by Kintsch (Reference Kintsch2001) to compute the intuitively plausible and contextually dependent vectors of propositions with the predicate argument structure. Given a proposition P(A), where P is a predicate and A is an argument (in the case of a noun compound, the modifier M is the predicate and the head H is the argument), the predication algorithm first chooses the m nearest neighbors of a predicate P, i.e., the m words with the highest cosine to P. The algorithm then selects k neighbors of P that are also related to A. Finally, the algorithm computes the centroid vector of P, A, and the k neighbors of P, and forms the vector representation of P(A). When the predication algorithm is applied to noun compounds, the set of neighbors of P relevant to A represents the intended sense of the modifier M that is appropriate for describing the intended sense of the head noun H.
Formally, the predication algorithm computes a compound vector p by the following formula:
\begin{eqnarray}
{\bf $p$} &=& \frac{{\bf $m$} + {\bf $h$} + \sum _{i=1}^k {\bf $q_i$}}{k+2}
\end{eqnarray}
For example, Table 1 shows how the predication algorithm calculates the composition vector of our example vectors h and m when m = 5 and k = 2. The leftmost column of Table 1 lists the neighbors of the modifier vector m in descending order of cosine similarity, whose values are listed in the second column. The predication algorithm first chooses the five nearest neighbors w5, w3, w1, w2, w9 (because m = 5), and then selects from them the two neighbors with the highest cosine to the head vector h (because k = 2). As shown in the third column, the vector w2 has the highest cosine to h and the vector w3 has the second highest cosine; thus, they correspond to q1 and q2 in (15). As a result, the predication algorithm yields the composition vector p = m + h + q1 + q2 = (1, 8, 4, 4, 0) + (0, 10, 4, 1, 6) + (2, 7, 7, 4, 5) + (2, 10, 9, 7, 7) = (5, 35, 24, 16, 18).
Example of the step-by-step behavior of the predication algorithm for the hypothetical vectors
3.2.5 Comparison
The predication algorithm does not take fully into account the relevance (or similarity) between the head noun and its intended sense in a noun compound. It is quite likely that a more plausible vector can be computed by using the set of common neighbors of the head and modifier, i.e., words that are semantically related to both the head and modifier.
This study proposes that we can take fully into account the relevance by applying Utsumi’s Reference Utsumi2011 comparison algorithm, which chooses the k common neighbors with the higher cosine both to the head and modifier, and then computes the centroid of these common vectors and the head vector. The comparison algorithm can be seen as a computational model of Gentner’s Reference Gentner1983 comparison (or analogical mapping) process consisting of alignment and projection Utsumi (Reference Utsumi2011). In addition, the effectiveness of the comparison algorithm as a vector composition method for noun compounds may be supported by the empirical findings of Wisniewski (Reference Wisniewski1996) stating that the comprehension of noun compounds involves such a comparison process.
Formally, the comparison algorithm computes a compound vector p using the following formula:
\begin{eqnarray}
{\bf $p$} &=& \frac{{\bf $h$} + \sum _{{\bf $c_i$}\,\in \,S} {\bf $c_i$}}{k+1}.
\end{eqnarray}
Example of common neighbors computed by the comparison algorithm for the hypothetical vectors
3.2.6 Dilation
The dilation algorithm was proposed by Mitchell and Lapata (Reference Mitchell and Lapata2010) as a basis-independent method for stretching or ‘dilating’ the head (or argument) vector along the direction of the modifier (or predicate) vector. This is achieved by decomposing the head vector h into two orthogonal vectors (i.e., a vector x parallel to m in (17) and a vector y orthogonal to m in (18)), and then stretching the parallel component x so that the modified vector p of h is more like m:
\begin{eqnarray}
{\bf $x$} &=& \frac{{\bf $m$}\cdot {\bf $h$}}{{\bf $m$}\cdot {\bf $m$}} {\bf $m$}
\end{eqnarray}
\begin{eqnarray}
{\bf $y$} &=& {\bf $h$} - {\bf $x$} = {\bf $h$} - \frac{{\bf $m$}\cdot {\bf $h$}}{{\bf $m$}\cdot {\bf $m$}} {\bf $m$}
\end{eqnarray}
\begin{eqnarray}
{\bf $p$} &=& \lambda {\bf $x$} + {\bf $y$} = (\lambda -1)\frac{{\bf $m$}\cdot {\bf $h$}}{{\bf $m$}\cdot {\bf $m$}} {\bf $m$} + {\bf $h$}
\end{eqnarray}
\begin{eqnarray}
{\bf $p$} &=& (\lambda -1)({\bf $m$}\,\cdot \,{\bf $h$})\ {\bf $m$} + ({\bf $m$}\,\cdot \,{\bf $m$})\ {\bf $h$}.
\end{eqnarray}
4 Evaluation experiment
4.1 Materials
The evaluation experiment was conducted in two languages, i.e., English and Japanese, to test whether different languages show consistent results.
In the experiment, we used two kinds of language resources, namely, a corpus and a thesaurus. Corpora were used to collect noun compounds and construct semantic spaces, while thesauri were used to collect unfamiliar noun compounds and to identify the meanings of words and compounds that are necessary for constructing test data. In this study, the written and non-fiction parts of the British National Corpus (henceforth BNC), and English WordNet 3.0 were used as an English corpus and thesaurus, respectively. Similarly, Japanese newspaper corpora (i.e., four years’ worth of Mainichi newspaper articles and two years’ worth of Nikkei newspaper articles), and the Japanese thesaurus ‘Nihongo Dai-Thesaurus,’Footnote 4 Yamaguchi (Reference Yamaguchi2006) were used as Japanese language resources.
4.1.1 Semantic spaces
Two semantic spaces, namely LSA-based and PPMI-based spaces, were constructed from these corpora. In order to exclude short documents that may adversely affect the quality of the constructed semantic spaces, we used as a unit of documents, English paragraphs that included 40 or more words and Japanese paragraphs that included 30 or more words. As a result, the English corpus contained 54.74 million words (73,422 types of words) and 491,106 documents (i.e., paragraphs), while the Japanese corpus contained 26.20 million words (63,875 word types) and 604,058 documents, as shown in Table 3.
Semantic spaces used in this study
Note. The columns of ‘word–doc’ and ‘word–word’ show the matrix densities of the word–document matrix for the LSA-based space and the word–word matrix for the PPMI-based space, respectively.
The LSA-based semantic space was constructed from the word–document matrix A whose dimensions were 73,422 × 491,106 for English and 63,875 × 604,058 for Japanese. Following previous work Mitchell and Lapata (Reference Mitchell and Lapata2010), we determined the number of dimensions n of the semantic space using a word-based similarity computation task. The task was identical to the task of related word ranking used in the evaluation experiment, which will be explained in Section 4.2.1, except that word–word pairs were used instead of compound–word pairs. One thousand English word–word pairs were selected from the University of South Florida Free Association Norms Nelson et al. (Reference Nelson, McEvoy and Schreiber1998) and 1,000 Japanese word–word pairs were selected from a Japanese word association norm ‘Renso Kijunhyo’ Umemoto (Reference Umemoto1969). We performed the task with different dimensions ranging from 100 to 2,000 in steps of 100 and compared the performance separately for English and Japanese word–word pairs. Both English and Japanese semantic spaces achieved the best performance when the number of dimensions was 300. Hence, 300-dimensional LSA spaces were used in the evaluation experiment.
The PPMI-based semantic space was constructed from the word–word matrix A with the same number of rows and columns (i.e., 73,422 × 73,422 for English, and 63,875 × 63,875 for Japanese), because in this study the word–word matrix was not smoothed. A context window of size 5 (i.e., five words on either side of the target word) was used for calculating the word co-occurrence frequency.
The density of the word–document or word–word matrix in Table 3 shows the percentage of non-zero elements of the matrix (after weighting), which is formally defined as (the number of non-zero elements) / (l×n) × 100. Given that the density of the initial matrix affects the quality of the semantic space, the English and Japanese semantic spaces may be equal in quality, because the two matrices are almost equal in density.
4.1.2 Compound-word pairs
In the evaluation experiment, we used two types of noun compounds, familiar compounds that occur in the corpus from which semantic spaces are constructed, and unfamiliar compounds that do not occur in the corpus. We used familiar compounds to evaluate how well composition algorithms can explicitly extract semantic information of noun compounds implicitly involved in the semantic space. Conversely, unfamiliar compounds were used to evaluate how well composition algorithms predict the semantic representation of noun compounds from semantic information of constituent words. Note that the ability to interpret unfamiliar compounds is particularly important for the algorithms not only as an NLP technique but also as a cognitive model, because people can easily interpret unfamiliar or novel noun compounds Wisniewski (Reference Wisniewski1996).
We chose familiar noun compounds that occur at least twenty times in the corpus and are also included in the thesaurus. Similarly, we selected unfamiliar noun compounds that do not occur in the corpus but are included in the thesaurus. All noun compounds comprised only two nouns (i.e., the head and modifier noun). As a result, 1,200 English and 2,191 Japanese familiar compounds, and 943 English and 427 Japanese unfamiliar compounds were chosen for evaluation.
For each of these chosen noun compounds, two semantically related words were selected from the English WordNet synsets (i.e., sets of synonyms) or the semantic categories of the Japanese thesaurus that include the target compound. If none or only one word could be selected, then related words were selected randomly from hypernyms or coordinate words of the target compound. (This additional selection procedure was not necessary for Japanese words, because the semantic categories of the Japanese thesaurus are represented by fewer levels than in the English WordNet, and thus contain more words as synonyms.) In this procedure, the head and modifier themselves were prohibited from being selected as related words. As a result, this procedure generated 2,166 English and 4,296 Japanese compound-word pairs for familiar compounds, and 1,698 English and 833 Japanese pairs for unfamiliar compounds. Table 4 shows some examples of pairs of a noun compound and its related word used in the evaluation experiment.
Examples of noun compounds and their related words used in the evaluation
Note. The original Japanese expressions are shown in parentheses, preceded by their literal English translations.
Finally, these compound-word pairs were classified as emergent or non-emergent by the following criterion:
• A compound-word pair is judged to be emergent if semantic categories – synsets, hypernyms, hyponyms, and coordinates of the English thesaurus or the deepest categories of the Japanese thesaurus – of the paired word include neither the head nor the modifier of the compound.
Simply put, this criterion means that the word of a compound-word pair expresses the emergent feature (or meaning) of the compound if both the head and modifier of the compound are unrelated to the paired word. By this criterion, 276 English and 1,037 Japanese pairs for familiar compounds and 277 English and 292 Japanese pairs for unfamiliar compounds were judged to be emergent. Furthermore, if a noun compound had none or only one emergent pair, additional emergent pair(s) were generated using the same procedure as explained above. As a result, 534 English and 2,032 Japanese emergent pairs for familiar compounds (including 315 English and 1,055 Japanese compounds) and 542 English and 643 Japanese emergent pairs for unfamiliar compounds (including 313 English and 333 Japanese compounds) were generated. Table 4 shows some examples of emergent and non-emergent pairs. It must be noted here that emergence is not a property for a noun compound alone, but a property of a compound-word pair. Hence, the same noun compound (e.g., ‘turning point’ in Table 4) appears in both emergent and non-emergent pairs. Note also that this emergence judgment procedure provides only an approximation of feature emergence that involves a more complex phenomenon.
In order to confirm the appropriateness of this emergence judgment procedure, we conducted an experiment in which human ratings of semantic relatedness were collected. In the experiment, we chose 25 emergent and 25 non-emergent compound-word pairs from the 4,296 Japanese compound-word pairs generated above. Fourteen participants were asked to rate the semantic relatedness between the compound and the word in all 50 compound-word pairs (e.g., semantic relatedness between ‘heart disease’ and ‘hypertension’) on a seven-point scale of one (unrelated) to seven (related). (These data were also used for the evaluation experiment in Section 4.2.) They were also asked to rate the semantic relatedness of the head or the modifier of the compound to the word paired with the compound (e.g., semantic relatedness between ‘disease’ and ‘hypertension’ or between ‘heart’ and ‘hypertension’). The presentation order of those pairs was randomized for each participant. If the emergence judgment procedure using a thesaurus is appropriate, it is expected that, in the case of an emergent compound-word pair, the paired word would be rated as semantically more related to the compound than to the constituent words (i.e., the head and modifier) of the compound. On the other hand, non-emergent compound-word pairs are not expected to exhibit such differences. The results of the evaluation experiment were consistent with the prediction. We compared the relatedness rating of a compound with the relatedness rating of the constituent words of that compound. The relatedness rating value of constituent words was determined as the higher rating between the head and modifier ratings. For emergent pairs, the relatedness rating of compounds to the paired words (M = 5.27) was significantly higher than the relatedness rating of constituent words (M = 4.99), t(24) = 2.65, p<.01. For non-emergent pairs, the relatedness rating of compounds (M = 5.96) was not significantly different from the relatedness rating of constituent words (M = 6.15), t(24) = 1.48, p>.10. These results confirm that our classification of emergent and non-emergent pairs is appropriate for an approximation of feature emergence.Footnote 5
4.2 Method
To evaluate which composition algorithms are appropriate for computing the attributional similarity of noun compounds, this study employs the following three tests for noun compounds: related word ranking, similarity correlation, and semantic classification. The first two tests (i.e., related word ranking and similarity correlation) are used to evaluate the relative performance of similarity computation, while the semantic classification test is designed to explore the absolute performance of the similarity computation.
4.2.1 Related word ranking
This test examines the degree of attributional similarity between the compound and its related word of a compound-word pair by finding the rank of the related word under the ordering imposed by the cosine. This kind of word ranking has been used as a performance test by many studies on semantic spaces Griffiths et al. (Reference Griffiths, Steyvers and Tenenbaum2007); Baroni and Zamparelli (Reference Baroni and Zamparelli2010).
Figure 1 illustrates the overall procedure of the word ranking test. First, for each compound-word pair (ci, wi) comprising a noun compound ci and its related word wi in a given set of pairs P described in Section 4.1.2, the vector for the noun compound ci was computed by a given composition algorithm. Afterward, the rank ri of the paired word wi was assessed by computing the cosine similarity between (the vector for) ci and (the vectors for) all words (including wi) in a semantic space, and sorting all words in descending order of the cosine. A higher rank implies that the word wi is semantically more related to the compound ci. For example, in the case of the compound-word pair (c 1, w 1) = (heartdisease, hypertension) shown in Figure 1, the word ‘hypertension’ has the eighth highest cosine similarity to ‘heart disease’ in the semantic space. Hence, for the pair (c 1, w 1), the rank r 1 is 8. Similarly, for the pair ‘love affair–romance’ the rank ri of the word ‘romance’ is 3. Next, all compound-word pairs in the set P were sorted in ascending order of the rank ri, as depicted on the right side of Figure 1. The sorted list of the rank ri is denoted as r′1, …, r′|P|. (Hence, for example, r′1 is the minimum rank in r 1, …, r |P|.) In Figure 1, the pair ‘love affair–romance’ ranks fifth (i.e., r′5 = (therankr1ofthatpair) = 3), while the pair ‘heart disease–hypertension’ ranks twentieth (i.e., r′20 = 8).
Overall procedure of the task of ranking related words.
Finally, the overall performance of each composition algorithm was measured by the median rank of the sorted list r′1, …, r′|P| for the set of pairs P. Formally, the median rank rmed(P) of a set P of compound-word pairs is defined as
\begin{equation}
r_{med}(P) = \left\lbrace \begin{array}{@{}l@{\quad }l@{}}r^{\prime }_{\frac{|P|+1}{2}} & \text{if $|P|$ is odd} \\
\frac{1}{2} \big ( r^{\prime }_{\frac{|P|}{2}} + r^{\prime }_{\frac{|P|}{2}+1} \big ) & \text{if $|P|$ is even.} \end{array}\right.
\end{equation}
4.2.2 Similarity correlation
In this test, we used correlation analysis to examine how well composition algorithms (and their computed similarity values) predict human similarity judgment.
In order to collect a manageable size of human judgment data, we chose one hundred Japanese compound-word pairs that comprised twenty-five non-emergent and twenty-five emergent pairs for familiar compounds, and twenty-five non-emergent and twenty-five emergent pairs for unfamiliar compounds.Footnote 6 We then asked fourteen participants, who were all native speakers of Japanese, to rate the semantic relatedness between the compounds and words of the 100 pairs stated above. These pairs were rated on a seven-point scale ranging from one (unrelated) to seven (related). The presentation order of those pairs was randomized for each participant. The mean ratings for each pair were used in the correlation analysis.
The performance of each composition algorithm was measured by Spearman’s correlation coefficient. First, for each of the one hundred compound-word pairs, the vector for the compound was generated by a given algorithm. Next, the cosine similarity between the compound vector and the vector for the paired word was calculated. Lastly, a correlation coefficient was calculated between the computed cosine values and mean human ratings of the compound-word pairs.
4.2.3 Semantic classification
In this test, each compound of the compound-word pairs generated in Section 4.1.2 was paired with a semantically unrelated word. During the test, a given set of pairs was classified according to the cosine similarity into two groups, namely semantically related and unrelated pairs. (Hence, this test can be regarded as a binary classification problem.)
Semantically unrelated pairs were constructed by combining the compound (e.g., ‘love affair’) of semantically related pairs (e.g., ‘love affair–romance’) with an unrelated (or less related) word (e.g., ‘finding’). Unrelated words were chosen randomly from the words that were unrelated to the compound, the head noun, and the modifier noun. Specifically, words were judged to be unrelated if none of their synsets, hypernyms, hyponyms, and coordinates included the target compound, head noun, and modifier noun for English tests, or if none of their basic categories included the target compound, head noun and modifier noun for Japanese tests. In addition, to largely reduce the undesirable effect of the word frequency on the cosine similarity, all unrelated words were chosen to occur more than fifty times in the corpus. Hence, the number of unrelated pairs generated was equal to the number of related pairs. As a result, the number of pairs that had to be classified was twice the number of compound-word pairs used in the word ranking test. For example, 4,332 English and 8,592 Japanese pairs were used for familiar compounds, and 3,396 English and 1,666 Japanese pairs were used for unfamiliar compounds.
In the evaluation experiment, all compound-noun pairs were sorted in descending order of cosine similarity, and the top k pairs were retrieved as semantically related. The performance for this binary classification was measured by average precision, which has been widely used for evaluating the ranked retrieval results Manning et al. (Reference Manning, Raghavan and Schütze2008). Average precision (AP) is the average of the precision value obtained for each recall level (i.e., after each ‘related’ pair is retrieved). For a ranked set of compound-noun pairs P, the average precision AP(P) calculated by
\begin{equation}
\text{AP}(P) = \frac{1}{N}\sum _{i=1}^{N} \text{Precision}(P_i)
\end{equation}
4.3 Results
For each of the six composition algorithms, both versions (i.e., all and emergent compounds) of the three tests were conducted in both English and Japanese and using LSA-based and PPMI-based semantic spaces, and their performance measures were calculated.
In order to compute the performance of the predication, comparison, and dilation algorithms, we estimated the optimal parameters using a tenfold cross-validation procedure. The cosine similarity (and the rank) of compound-word pairs in each group was calculated, with the optimal parameters estimated using the other nine groups as training data. Optimal parameters were estimated based on the median rank for the word ranking and similarity correlation tests, while they were estimated based on the mean average precision (henceforth MAP) over training groups for the semantic classification task. The parameter space was given such that the parameter m (for predication) varied between 1 and 50 and between 100 and 500 in steps of 50, the parameter k (for predication and comparison) varied between 1 and 10, and the parameter λ (for dilation) varied between 1.1 and 10.0 in steps of 0.1. Note that we removed any combinations of m and k for which m<k.
Furthermore, for purpose of comparison, two non-compositional methods were considered for obtaining compound vectors. The first method, regarded as the baseline, simply used the vector representing the head noun as the compound vector. (Note that for λ = 1.0, the dilation algorithm is identical to this baseline algorithm.) The second method computed compound vectors directly from the corpus; in other words, semantic spaces were constructed in which all the target noun compounds were added as single words. This method was expected to achieve better and probably the best performance, which suggests a target level for the composition algorithms. Note that this method could not be applied to unfamiliar compounds because they were not included in the corpus.
4.3.1 Overall performance
In this section, the results of the three tasks for familiar noun compounds are shown. The results of the unfamiliar noun compounds will be shown later in Section 4.3.4.
Related word ranking. Tables 5 and 6 respectively show the results (i.e., median rank) of the word ranking test when LSA-based and PPMI-based semantic spaces are used for vector composition. In addition, Figures 2 and 3 also show the first and third quartile ranks of the ranking of a word, as well as the median ranks. (Because the third quartile rank is much lower than the median rank and the first quartile rank, it is not used as a performance measure in this paper.)
Boxplots of the rank of semantically related words to familiar noun compounds in the ordering computed in the LSA-based semantic space. The bottom and top of the box represent the first and third quartile ranks, respectively, and the band near the middle of the box represents the median rank. Note that a lower rank value implies better performance.
Boxplots of the rank of semantically related words to familiar noun compounds in the ordering computed in the PPMI-based semantic space.
Median rank of words semantically related to familiar noun compounds in the ordering computed in the LSA-based semantic space
Note. Underlined numbers indicate the lowest median rank values (i.e., the best performance) among the composition algorithms. Numbers in parentheses denote the ranks of the algorithms in ascending order of median rank.
Median rank of words semantically related to familiar noun compounds in the ordering computed in the PPMI-based semantic space
Note. Underlined numbers indicate the lowest median rank values (i.e., the best performance) among the composition algorithms. Numbers in parentheses denote the ranks of the algorithms in ascending order of median rank.
Concerning the results of the LSA-based semantic space (shown in Table 5 and Figure 2), it is not surprising that the original compound vector computed directly from the corpus yields the best performance (i.e., both median rank and first quartile rank) for Japanese compounds. In the case of English compounds, the original compound vector also yields higher performance. Among the six composition algorithms, the dilation algorithm achieves the best performance in both languages, although the comparison algorithm achieves the best performance for English emergent pairs. In addition, only the dilation and comparison algorithms perform better than the head noun vector (i.e., baseline) in both languages and in both types of pairs. These results suggest that the dilation and comparison algorithms may be effective in computing the meaning of noun compounds. Note that the circular convolution and multiplication algorithms have much worse performance than the other four algorithms.
The results of the PPMI-based semantic space (shown in Table 6 and Figure 3) are radically different from those of the LSA-based space in one point. The multiplication algorithm achieves the highest performance among all the algorithms in the PPMI-based semantic space, even though it achieves the lowest performance when the LSA-based semantic space is used. These opposing results clearly indicate that the performance of the multiplication algorithm depends heavily on the characteristics of semantic spaces. Possible reasons for these results will be discussed in Section 5. The relative performances of all composition algorithms other than multiplication do not differ largely between the PPMI-based and LSA-based semantic spaces. The dilation algorithm performs better than the other algorithms across languages and types of pairs. The comparison algorithm performs worse in the PPMI-based space than in the LSA-based space; it underperforms the baseline in both types of pairs, and the predication and centroid algorithms in the case of all pairs. Circular convolution also achieves much worse performance.
Concerning effects that may occur due to the use of different languages, the relative performances among the composition algorithms do not differ between English and Japanese. This consistency in English and Japanese results increases the generality of the findings in this paper. However, note that a difference is observed regarding the absolute ranks; as shown in Figures 2 and 3, the median and first quartile ranks of Japanese compound-word pairs are lower (thus better) than those of English pairs (i.e., the boxes for Japanese compound-word pairs are positioned lower than the boxes for English pairs). One possible reason for this difference may be that the difficulty levels for English and Japanese tests are different, owing to the fact that different kinds of thesauri are used for English and Japanese. As mentioned in Section 4.2, the distance in the hierarchical structure of the thesaurus determines which words are semantically related to the target compounds, and thus different thesauri may generate test items with different difficulty levels.
Similarity correlation. Table 7 shows correlation coefficients between human similarity rating and cosine similarity computed in both semantic spaces.
Correlation coefficients between human similarity ratings and the cosine similarity computed by the composition algorithms for Japanese familiar compounds
Note. Underlined numbers indicate the highest coefficients among the composition algorithms. Numbers in parentheses denote the ranks in descending order of the correlation coefficients.
† p<.10. *p<.05. **p<.01.
Let us first consider the result of all compound-word pairs (n = 50) shown in the second and third columns of Table 7. When the LSA-based semantic space is used, the four composition algorithms (i.e., centroid, predication, comparison, and dilation) are significantly correlated with human judgments, while the multiplication and circular convolution algorithms are not. The dilation algorithm outperforms all other algorithms including the original compound vector. These results are almost consistent with the results of the word ranking test. The results of the correlation analysis of the PPMI-based semantic space are also consistent with the results of the word ranking test. The multiplication algorithm shows a significant correlation with human judgment and performs better than all other algorithms except predication. The high performance of the predication algorithm is not observed in the word ranking test. The other three composition algorithms (i.e., dilation, comparison, and centroid) are significantly correlated with human judgments, but the circular convolution algorithm is not.
Correlations calculated for emergent pairs (n = 25) are not statistically significant, and thus it may not be appropriate to derive a reliable conclusion from the results of the emergent pairs. However, the results of the emergent pairs are almost consistent with the results of all the pairs; the correlation coefficients of the dilation and multiplication algorithms are relatively high in the LSA-based and PPMI-based semantic spaces, respectively. A notable difference is that the correlation coefficient of circular convolution is rather high and marginally significant in the LSA-based space.
Semantic classification. Tables 8 and 9 respectively show the results (i.e., mean average precision) of the semantic classification test in the LSA-based and PPMI-based semantic spaces.
Mean average precision of semantic classification for familiar compounds using the LSA-based semantic space
Note. Underlined numbers indicate the highest MAP values among the composition algorithms. Numbers in parentheses denote the ranks of the algorithms in descending order of the MAP values.
Mean average precision of semantic classification for familiar compounds using the PPMI-based semantic space
Note. Underlined numbers indicate the highest MAP values among the composition algorithms. Numbers in parentheses denote the ranks of the algorithms in descending order of the MAP values.
On the whole, the results of semantic classification are consistent with the results of the word ranking and similarity correlation tests. When the LSA-based semantic space is used, the dilation algorithm performs consistently well; its MAP value is higher than the other composition algorithms in the case of all pairs, and also higher than the original vector for emergent pairs. The comparison algorithm achieves the best performance particularly for emergent pairs; this result is partially consistent with the result of the word ranking test. Only the dilation and comparison algorithms perform better than the baseline, regardless of language and type of pair. The multiplication and circular convolution algorithms perform worse than the other algorithms. In particular, the multiplication algorithm performs very poorly. In the PPMI-based semantic space, the multiplication algorithm achieves the best performance, regardless of language and type of pair, as in the case of the word ranking test. The dilation algorithm also performs consistently well, although its MAP value is lower than that of the original vector when Japanese pairs are considered. The comparison algorithm performs somewhat worse than in the LSA-based space; it underperforms not only the baseline, but also the predication and centroid algorithms. The circular convolution algorithm has the lowest performance, as in the case of the word ranking test.
Summarizing, based on the above results, we conclude that the dilation algorithm is generally most effective in computing the meaning of noun compounds in the LSA-based semantic space, while the multiplication algorithm is best suited to compute the meaning of noun compounds in the PPMI-based semantic space. In particular, the finding that these algorithms can exceed the performance obtained by the original vector suggests that vector composition not only approximates the original vector but also actively elicits (or mines) the implicit knowledge of compound meanings behind the corpus. Furthermore, these results are not specific to a certain language, thus indicating some generality of the findings.
4.3.2 Effect of frequency of noun compounds
To assess to what degree an algorithm’s performance is affected by the frequency of the compounds in the corpus, we also examined the performance scores (i.e., median rank and mean average precision) calculated when the target compounds were limited to those that occur at least tf times in the corpus. Figures 4 and 5 show the median rank of the word ranking test for a threshold tf ranging from 20 (i.e., ‘unlimited’) to 400 in steps of 10. Graphs of mean average precision for the semantic classification test calculated with the same thresholds are shown in the Appendix (i.e., Figures 11 and 12). (Note that the graphs for the multiplication and circular convolution algorithms in the LSA-based space, and graphs for the circular convolution algorithm in the PPMI-based space are not shown in these figures because their performance is much lower than that of the other algorithms regardless of frequency.)
Median rank as a function of minimum frequency of noun compounds in the case of the LSA-based semantic space.
Median rank as a function of minimum frequency of noun compounds in the case of the PPMI-based semantic space.
As expected, both median rank and mean average precision are proportionally dependent on the frequency threshold tf; they get better (i.e., lower rank value and higher MAP value) as target compound-word pairs are limited to more frequent compounds. One plausible explanation for the result is that frequent compounds are used as if they are lexicalized, and thus the vectors for the constituent words (i.e., head and modifier) of frequent compounds implicitly involve the semantic knowledge for the compounds, which can be accurately extracted by the composition algorithms. On the other hand, the constituent words of less frequent compounds are not likely to involve the semantic information of the compounds.
However, one unexpected result is obtained that the performance of some compound-word pairs drops for highly frequent compounds. For example, Figure 4 depicts that the performance of all compound-word pairs in the LSA-based space drops (i.e., the median rank rises) for compounds with corpus frequency >150 (English) and >180 (Japanese). This tendency is less pronounced in the PPMI-based space, as shown in Figure 5, but the performance of English pairs becomes worse for compounds with corpus frequency >350. One possible explanation is that highly frequent compounds (and their constituent words) are semantically related to a number of words, and consequently it is more difficult for a semantic space model to compute the vectors of highly frequent compounds that accurately grasp the meaning of the compounds than the vectors of less frequent compounds.
Concerning whether the relative performances of the composition algorithms change over frequency, no drastic changes are observed, as shown in Figures 4 and 5 or in Figures 11 and 12. Regardless of the frequency threshold, the dilation algorithm performs consistently better than the other algorithms (excluding the original vector) in the LSA-based semantic space, while the multiplication algorithm outperforms the other algorithms in the PPMI-based semantic space. This result suggests that the compound frequency does not explain the observed performance differences among the composition algorithms.
4.3.3 Effect of constituent frequency
The quality of a composition vector depends not only on compound frequency, as shown in Section 4.3.2, but also on the frequency of constituent words (i.e., head and modifier), because word frequency has a great effect on the quality of word vectors. Hence, in this section, we examine the relationship between constituent frequency and the performance of composition algorithms.
Figures 6 and 7 show the median rank of all compound-word pairs for a threshold tf of constituent frequency ranging from 10 to 104. (Note that the data for high constituent frequency are not plotted in some graphs because their values of median rank are too high to be plotted.) The performance of composition algorithms slightly increases (i.e., the median rank decreases) in the case of the LSA-based space, or does not change in the case of the PPMI-based space, regardless of whether head frequency or modifier frequency is considered. This result suggests that constituent frequency has little effect on the performance of composition algorithms. The quality of composition algorithms may depend on the degree to which the vectors for the constituent words implicitly involve the semantic knowledge of the compounds, but such the degree may have nothing to do with constituent frequency (and depends on compound frequency).
Median rank of all compound-word pairs as a function of minimum frequency of constituent words (head or modifier) in the case of the LSA-based semantic space.
Median rank of all compound-word pairs as a function of minimum frequency of constituent words (head or modifier) in the case of the PPMI-based semantic space.
In addition, the performance drops for compounds with very high constituent frequency (i.e., about 103 or more), as in the case of the relation between compound frequency and algorithms’ performance described in Section 4.3.2. The reason may be the same; highly frequent words are semantically related to many words, and thus it is more difficult for a composition algorithm to extract the specific meaning appropriate for noun compounds.
4.3.4 Performance for unfamiliar noun compounds
In this section, we show the results of the three tests for unfamiliar compounds, and compare them with those of familiar compounds.
Related word ranking. Tables 10 and 11 show the results of the word ranking test for unfamiliar compounds in the LSA-based and PPMI-based semantic spaces. Figures 8 and 9 show boxplots of the ranks (i.e., the first and third quartile ranks as well as the median rank) of semantically related words to unfamiliar compounds. (Note again that because unfamiliar compounds are not included in the corpus, their original vector cannot be computed.)
Boxplots of the rank of semantically related words to unfamiliar noun compounds in the ordering computed in the LSA-based semantic space. The bottom and top of the box represent the first and third quartile ranks, respectively, and the band near the middle of the box represents the median rank. Note that a lower rank value implies better performance.
Boxplots of the rank of semantically related words to familiar noun compounds in the ordering computed in the PPMI-based semantic space. The bottom and top of the box represent the first and third quartile ranks, respectively, and the band near the middle of the box represents the median rank. Note that a lower rank value implies better performance.
Median rank of words semantically related to unfamiliar noun compounds in the ordering computed in the LSA-based semantic space
Note. Underlined numbers indicate the lowest median ranks (i.e., the best performance) among the composition algorithms. Numbers in parentheses denote the ranks of the algorithms in ascending order of the median rank.
Median rank of words semantically related to unfamiliar noun compounds in the ordering computed in the PPMI-based semantic space
Note. Underlined numbers indicate the lowest median rank values (i.e., the best performance) Numbers in parentheses denote the ranks of the algorithms in ascending order of the median rank.
The performance of unfamiliar compounds still remains at a high level, although lower than the performances of familiar compounds (as compared with Tables 5 and 6). This result demonstrates the ability of semantic space models to comprehend unfamiliar compounds.
As shown in Tables 10 and 11, and in Figures 8 and 9, the relative performances of the composition algorithms do not change noticeably between familiar and unfamiliar compounds. In the LSA-based semantic space, regardless of whether compounds are familiar or unfamiliar, the dilation algorithm shows the best performance (i.e., lowest first quartile rank in all cases and lowest median rank in the case of all pairs). Only two algorithms, dilation and comparison, perform better than the baseline regardless of language and type of pair. On the PPMI-based semantic space, the multiplication algorithm also performs consistently better than the other algorithms in terms of first quartile rank. Among the other algorithms, the dilation algorithm performs best as in the case of familiar pairs. The consistency of these results suggests that the results obtained are intrinsic to vector composition algorithms.
However, it must be noted that some notable differences are observed between the results of familiar and unfamiliar compounds. When the LSA-based space is used, the comparison algorithm achieves a lowest median rank for emergent pairs and a lowest first quartile rank for Japanese emergent pairs. This result may suggest that the comparison algorithm is more appropriate for representing emergent meanings of unfamiliar compounds. Another difference is that the performance of the multiplication algorithm in the PPMI-based space seems to be lower for unfamiliar compounds; the dilation algorithm has a lower median rank than the multiplication algorithm for Japanese pairs, and the predication algorithm has a lower median rank for English emergent pairs.
Similarity correlation. Table 12 shows the correlation coefficients between human similarity rating and cosine similarity computed in both semantic spaces.
Correlation coefficients between human similarity ratings and the cosine similarity computed by the composition algorithms for Japanese unfamiliar compounds
Note. Underlined numbers indicate the highest coefficients among the composition algorithms. Numbers in parentheses denote the ranks of the algorithms in descending order of the correlation coefficients.
† p<.10. *p<.05. **p<.01.
The results of the correlation analysis for unfamiliar noun compounds are also essentially the same as the results for familiar compounds; when all compound-word pairs are considered, the three composition algorithms (i.e., centroid, comparison, and dilation) are significantly correlated with human judgment in both semantic spaces, and the multiplication algorithm is significantly correlated only in the PPMI-based space. In addition, the dilation algorithm outperforms all other algorithms in the LSA-based semantic space, while the multiplication algorithm outperforms all other algorithms in the PPMI-based space. One difference is that the predication algorithm does not have a significant correlation with human judgment for unfamiliar compounds, although it has for familiar compounds.
Concerning emergent pairs, the result that no algorithms achieve a significant correlation with human judgment is common to both familiar and unfamiliar compounds, suggesting that emergent meanings of compounds are more difficult to represent in a semantic space model. In both spaces, the comparison algorithm has a higher correlation for unfamiliar compounds than for familiar compounds, while the dilation algorithm has a lower correlation for unfamiliar compounds. The multiplication algorithm shows an unexpectedly higher correlation in the LSA-based space.
Semantic classification. Tables 13 and 14 respectively show the results (i.e., mean average precision) of the semantic classification test for unfamiliar compounds using the LSA-based and PPMI-based semantic spaces.
Mean average precision of semantic classification for unfamiliar compounds using the LSA-based semantic space
Note. Underlined numbers indicate the highest MAP values among the composition algorithms. Numbers in parentheses denote the ranks of the algorithms in descending order of the MAP values.
Mean average precision of semantic classification for unfamiliar compounds using the PPMI-based semantic space
Note. Underlined numbers indicate the highest MAP values among the composition algorithms. Numbers in parentheses denote the ranks of the algorithms in descending order of the MAP values.
Overall performance of unfamiliar compounds also remains at a high level, although it is worse than the performance of familiar compounds. As in the case of the word ranking test, the relative performance of the composition algorithms does not differ greatly between familiar and unfamiliar compounds. The dilation algorithm yields consistently better performance in the LSA-based space, and the multiplication algorithm yields the best performance in the PPMI-based space. However, one notable difference is that the comparison algorithm performs best in more cases than for familiar compounds when the LSA-based space is used; the comparison algorithm performs best for Japanese compounds or emergent compounds. Taken together with the results of the word ranking test, this result suggests that the comparison algorithm may be more appropriate for deriving the meanings of unfamiliar compounds, which are neither explicitly nor implicitly involved in the semantic space, from the semantic knowledge of the constituent words involved in the semantic space.
5 Discussion
5.1 Implications of the experimental results
In this section, we discuss the characteristics of each composition algorithm that are implied from the results of the evaluation experiment.
5.1.1 Dilation algorithm and predication algorithm
The evaluation experiment demonstrated that the dilation algorithm shows consistently better performance than the other vector composition algorithms for computing the attributional similarity of noun compounds. This finding is consistent with the finding reported by Mitchell and Lapata (Reference Mitchell and Lapata2010) stating that the dilation algorithm performed consistently well when computing the similarities between noun compounds, as well as between adjective–noun combinations and between verb–object combinations. This finding suggests that an appropriate compound vector can be computed by moderately modifying the head vector. This also explains why the predication algorithm does not perform well, although it embodies the head–modifier interaction. The interaction in the predication algorithm is more modifier-directed and thus the predication algorithm is less likely to capture the relevance between the meanings of a noun compound and its head noun. (Hence, it can be presumed that the predication algorithm may be suitable for noun compounds in which the modifier plays a more important role in determining the compound meaning than the head noun.)
5.1.2 Multiplication algorithm
The multiplication algorithm, a simple multiplicative model with no free parameters, is found to perform quite differently depending on the type of semantic spaces; it performs best in the PPMI-based semantic space, while it performs worst in the LSA-based space. One possible reason for this sharply contrasting result lies in the meaningfulness of vector dimensions or components in a semantic space. The function of the component-wise multiplication is to highlight dimensions relevant to both vectors and downplay irrelevant dimensions. For example, in the case of the hypothetical vectors in (9), component-wise multiplication of the two vectors h = (0, 10, 4, 1, 6) and m = (1, 8, 4, 4, 0) results in (0, 80, 16, 4, 0), in which the second dimension is highlighted and the first and fifth dimensions are downplayed (or zeroed). Hence, if the dimensions of a semantic space represent distinctive features or meanings, component-wise multiplication is expected to behave as a meaningful function for the semantic interaction between words. In the PPMI-based semantic space, each component of the word vectors has a clear meaning; the jth component represents the number of co-occurrence with the context word wj, giving the semantics for component-wise multiplication of two word vectors. If two constituent words of a compound are likely to co-occur with the same context word, that compound is also likely to occur with that context word. If either of the constituent words do not co-occur with a certain context word, that compound is also unlikely to occur with that context word. Therefore, as demonstrated in Section 4, when the PPMI-based semantic space is used, the multiplication algorithm performs much better than the other algorithms for vector composition. On the other hand, if vector dimensions do not represent clear meanings or distinctive features in its own right, component-wise multiplication seems not to work as intended. The LSA-based semantic space corresponds to this case; dimensions of the low-dimensional space reduced by SVD do not have a clear semantics. For example, the fact that the ith component of an LSA-based vector x is zero does not necessarily imply that the ith component of a composition vector of x and y should be zero. In other words, we cannot judge whether the ith component of the composition vector should be zero or not, unless the ith dimension represents a certain distinctive feature or meaning. Therefore, the multiplication algorithm performs much worse than the other composition algorithms in the LSA-based semantic space. Note that our results for the multiplication algorithm are consistent with the findings reported by Mitchell and Lapata (Reference Mitchell and Lapata2010) finding that the multiplication algorithm performed best on a simple semantic space, because their semantic space was based on the word–context frequency matrix used in the PPMI-based space of this paper. (They did not use a PPMI weighting scheme, but both their and our spaces were not reduced by SVD and, thus, both dimensions had clear semantics.)
In order to justify our argument that the poor performance of the multiplication algorithm in the LSA-based space is due to the lack of dimension semantics, we conducted two additional experiments. In the first experiment, we compared the performance of the multiplication algorithm among eight semantic spaces, which were obtained from all combinations of two initial matrices (i.e., word–document matrix or word–word matrix), two weighting schemes (i.e., the product of log word frequency and the entropy defined in (1) and PPMI defined in (7)), and whether or not a matrix is smoothed by SVD. This experiment aims at testing whether the observed performance difference between the LSA-based and PPMI-based spaces is due to SVD smoothing or otherwise, because the initial matrix and the weighting scheme are also different between these two spaces. Table 15 shows the result of the word ranking test when eight semantic spaces were used for vector composition. Note that the combination of a word–document matrix, a logtf-entropy weighting method, and the presence of SVD smoothing (i.e., first row of Table 15) corresponds to the LSA-based space used for the evaluation experiment in Section 4, while the combination of a word–word matrix, a PPMI weighting method, and the absence of SVD smoothing (i.e., last row of Table 15) corresponds to the PPMI-based space. Table 15 clearly shows that SVD smoothing greatly reduces the performance of the multiplication algorithm regardless of the initial matrix and the weighting scheme. Therefore, it can be concluded from the result that the poor performance of the multiplication algorithm in the LSA-based space is attributed to SVD smoothing.
Comparison of the eight semantic spaces including the LSA-based and PPMI-based spaces in terms of median rank of words semantically related to familiar noun compounds
In SVD smoothing, an original word vector is projected onto the low-dimensional reduced space, and as a result an initial sparse matrix usually becomes dense. If a word vector has many zero dimensions, the multiplication algorithm seems to work well because it can downplay such the dimensions as irrelevant semantic information. Hence, it is uncertain whether the poor performance in the LSA-based space is due to the loss of dimension semantics (caused by dimensionality reduction of SVD smoothing) or simply due to sparsity reduction. The second additional experiment was conducted to examine which is the main cause of the poor performance of the multiplication algorithm. If sparsity reduction is the main cause, the performance of the multiplication algorithm would be improved simply by zeroing out small values in LSA-based word vectors. On the other hand, if the poor performance essentially comes from SVD smoothing, such the simple ‘zeroing out’ method would not improve the performance. In this experiment, we examined whether the performance of the multiplication algorithm is improved by zeroing out elements with smaller absolute values than a certain threshold. Figure 10 shows the median rank of the multiplication algorithm (and the centroid algorithm) in the sparsified LSA-based space where smaller values are zeroed out. In this figure, the horizontal axis denotes the percentage of zeroed elements (in other words, the degree of sparsity), and thus its value of zero indicates the original LSA-based space. The result was that, as shown in Figure 10, the performance of the multiplication algorithm was not improved by zeroing out LSA vectors, although a very slight improvement was observed in the Japanese space. This finding indicates that zeroing has little effect on the performance of the multiplication algorithm. It can thus be concluded that the poor performance of the multiplication algorithm is definitely due to the loss of dimension semantics by SVD smoothing.
Zeroing effect on the performance (i.e., median rank for familiar compounds) of the LSA-based semantic space.
5.1.3 Centroid algorithm
The centroid algorithm, a simple additive model, does not perform well overall; its performance is lower than the dilation algorithm. This result seems to suggest that the centroid algorithm is a less useful method for vector composition.
On the other hand, it is also a fact that the centroid algorithm has been widely used in semantic space models. One possible reason may lie in its robustness; as shown in Table 15, the performance of the centroid algorithm is rather stable independent of the type of semantic spaces, as compared to the multiplication algorithm. For example, the centroid algorithm achieves fairly good performance regardless of whether vector dimensions represent distinctive features or not.
5.1.4 Comparison algorithm
The performance of the comparison algorithm also depends on the type of semantic spaces and the compound familiarity. The overall tendency is that the comparison algorithm performs better in the LSA-based space than in the PPMI-based space; for example, the relative performance of the comparison algorithm shown in Tables 5 and 8 is better than that in Tables 6 and 9. In the LSA-based semantic space, the comparison algorithm is more appropriate for representing the meanings of unfamiliar compounds than for representing the meanings of familiar compounds. In some cases (especially in the case of emergent pairs shown in Tables 5, 10, and 13), the comparison algorithm outperforms the dilation algorithm. This finding suggests that common neighbors of the head and modifier, which are not taken into account by the dilation algorithm, provide a basis for representing or mining the implicit knowledge (i.e., emergent meanings) behind the corpus. In addition, the better performance for unfamiliar compounds (or emergent meanings) in the LSA-based space may be due to a smoothing effect; extracting the implicit knowledge benefits more from SVD smoothing, and the comparison algorithm may benefit more from the smoothing effect than other algorithms.
The effect of common neighbors on the computation of emergent meanings is also supported by the results of the optimal values of the parameter k (i.e., the number of common neighbors). Table 16 lists the optimal values and their averages of the parameter k estimated by the tenfold cross-validation procedure. This table clearly demonstrates that the optimal number of common neighbors is larger for the emergent pairs than for all pairs. This result indicates that more common neighbors are required for computing the emergent meanings than for computing the non-emergent meanings.
Optimal number of common neighbors estimated in the evaluation experiments using the tenfold cross-validation procedure
Note. Numbers in parentheses show the average numbers of common neighbors.
5.1.5 Circular convolution algorithm
The circular convolution algorithm performs consistently much worse than the other composition algorithms regardless of language, semantic space, and compound familiarity. This result of circular convolution is not surprising. As mentioned in Section 3.2, circular convolution is simply designed to bind together pairs of vectors so that the resulting vector can be decoded back into the original vectors using its syntactic (or word-order) information; it is not concerned with the process of semantic composition. Therefore, it is likely that the circular convolution algorithm is not adequate as a method for computing the meaning of noun compounds. Note that the experiment of Mitchell and Lapata (Reference Mitchell and Lapata2010) yielded the same result for the circular convolution algorithm.
5.1.6 Composition algorithm versus original vector
Interestingly, it is also found that in some cases (e.g., median rank and first quartile rank for English pairs shown in Table 5 and Figure 2) certain composition algorithms (especially the dilation and multiplication algorithms) outperform the non-compositional compound vector constructed directly from the corpus by treating noun compounds as single words. This finding is also consistent with the result of Mitchell and Lapata (Reference Mitchell and Lapata2010) that some compositional vectors performed better than the original non-compositional vectors. This finding indicates a high representational ability of the vector composition algorithms, particularly for moderate interactions between the head and modifier. It also suggests that, in general, the semantic space model provides a powerful framework for computational lexical semantics.
5.2 Compound disambiguation and semantic space model
This study has examined the ability of the semantic space model to compute the attributional similarity of a noun compound without considering semantic relations holding between the head and modifier. However, comprehending noun compounds is actually a more complicated process, in which semantic relations may have a large influence on similarity computation. For example, some types of relations may yield more emergent meanings than others, and such differences would affect the performance of similarity computations.
One promising way of utilizing semantic relations is to use words or phrases expressing the semantic relation (e.g., ‘made of’ and ‘cause’) when vectors for constituent words are combined for vector composition. For example, when a noun compound ‘apple pie’ is used to refer to ‘pie made of apple,’ the vector representation of ‘apple pie’ can be reasonably assumed to be the vector of its paraphrase ‘pie made of apple.’ To compute such a vector, first the predicate vector of ‘be made of apples’ is computed from the vectors for ‘be made of’ and ‘apple,’ and the sentence vector is then computed from the vectors of the argument ‘a pie’ and the predicate ‘be made of apples.’ (A similar approach is taken by Kintsch Reference Kintsch and Gibbs2008 for solving analogy problems.)
To this end, the process of identifying semantic relations is necessary. As mentioned in Section 1, computational methods for compound disambiguation have been extensively studied, and semantic space models can also be applied to identify semantic relations. For example, a number of studies have attempted to identify semantic relations by computing the attributional similarity between constituent nouns of two noun compounds (e.g., Rosario and Hearst Reference Rosario and Hearst2001; Rosario, Hearst and Fillmore Reference Rosario, Hearst and Fillmore2002; Nastase and Szpakowicz Reference Nastase and Szpakowicz2003; Kim and Baldwin Reference Kim, Baldwin, Dale, Wong, Su and Kwong2005; Ó Séaghdha and Copestake Reference Ó Séaghdha and Copestake2009). The rationale behind this method is that constituent words (i.e., head or modifier) of noun compounds involving the same semantic relation are likely to belong to the same category. For example, if one knows that the semantic relation ‘made-of’ holds true for noun compounds ‘apple pie’ and ‘strawberry jam,’ it is quite reasonable to assume that ‘orange juice’ encodes the same semantic relation, because these head nouns can be classified into the same semantic category ‘foods,’ and the modifiers can be classified as ‘fruits.’ A semantic space model can compute the attributional similarity between words, which can be used to classify each constituent word into an appropriate semantic category. Turney (Reference Turney2006) also addresses this problem within the framework of a semantic space model, but his method, the so-called latent relational analysis, computes the relational similarity using a pair–pattern matrix, which is different from a word–document matrix and a word–context matrix used in this study.
6 Conclusion
Through the evaluation experiment, this paper has shown the validity of a semantic space model for computing the attributional similarity of noun compounds. The findings obtained in this study are summarized as follows:
• The dilation algorithm is generally most effective in computing the attributional similarity of noun compounds. This suggests that the dilation algorithm is in general a versatile vector composition algorithm.
• The multiplication algorithm performs quite differently depending on the type of the semantic space. It is best suited for vector composition in PPMI-based (or word co-occurrence-based in general) semantic spaces. This suggests that component-wise multiplication is effective for semantic spaces whose dimensions represent distinctive features or semantics.
• The comparison algorithm is suited for representing the (emergent) meanings of unfamiliar noun compounds in the LSA-based semantic space. Its performance is equal or superior to the performance of the dilation algorithm; this suggests that the comparison algorithm is effective in eliciting the implicit knowledge of compound meanings.
• The predication and centroid algorithms are not, on the whole, superior to the dilation, multiplication, and comparison algorithms. This may be largely due to the lack of the moderate interaction between the head and modifier.
• The circular convolution algorithm shows much worse performance than any of the other composition algorithms. It is not appropriate for semantic composition.
• The relative performances among the vector composition algorithms considered in this study, do not significantly differ between English and Japanese tests. This indicates that the findings obtained are not specific to a particular language; they are likely to be common to any language.
It would be interesting and vital for further work to develop a method for computing the attributional similarities that utilizes semantic relations holding between the head and modifier, as well as a more efficient method for vector composition. Additionally, we are working toward extending this study to longer noun compounds consisting of three or more nouns.
Acknowledgments
This study was supported in part by Grants-in-Aid for Scientific Research C (No. 20500234) and Scientific Research B (No. 23300098) from the Japan Society for the Promotion of Science. I would like to thank three anonymous reviewers for their insightful comments and suggestions, which helped me improve the article. Of course, all remaining errors are of my own responsibility.
Appendix: Effect of frequency of noun compounds on the task of semantic classification
Figures 11 and 12 show the MAP values of the semantic classification test for a threshold tf ranging from 20 to 400 in steps of 10. These figures are discussed in Section 4.3.2.
Mean average precision of semantic classification as a function of minimum frequency of noun compounds in the case of the LSA-based semantic space.
Mean average precision of semantic classification as a function of minimum frequency of noun compounds in the case of the PPMI-based semantic space.
