1.1 Distributional models for linguistic information processing

Semantic spaces are usually built to capture contextual co-occurrences of words. They can be thus classified according to the types of such contexts. Different classes of contexts tend to emphasize different types of semantic properties, and thus semantic relations among words (Sahlgren Reference Sahlgren2006).

In document-based spaces, contexts are entire documents. These spaces have been historically used in Information Retrieval, and tend to capture topic similarity between words – i.e. words similar in the space tend to refer to the same topic, as they appear in similar documents (e.g. ‘doctor’/‘hospital’ for the medical topic).

Word-based spaces (e.g. Bullinaria and Levy Reference Bullinaria and Levy2007) define contexts as the words appearing in a n-window of the target (i.e. n tokens on the left and on the right). Such spaces model a generic notion of semantic relatedness. Two target words close in the space are likely to be related by some type of generic semantic relation, either paradigmatic (e.g. synonymy, hyperonymy, and antonymy) or syntagmatic (e.g. meronymy, conceptual association, and phrasal association). Indeed, words with similar co-occurrences can be both words occurring together in a text (e.g. ‘doctor’ and ‘patient’ in ‘the doctor visited the patient’, sharing the same contexts ‘visited’) and substitutional words (e.g. ‘doctor’ and ‘surgeon’ in ‘the (doctor-surgeon) visited the patient’). Sahlgren (Reference Sahlgren2006) and Erk and Padó (Reference Erk and Padó2008) present interesting analysis and insights on these classes of semantic spaces.

More recently, Pado and Lapata (Reference Pado and Lapata2007) introduced syntax-based spaces, where contexts correspond to lexical syntactic relations (e.g. X-VSubj-doctor for every target word X). These spaces are very effective in modeling semantic similarity. Two target words close in the space are likely to be in a paradigmatic relation, i.e. to be categorically close (as in a is-a hierarchy, Budanitsky and Hirst Reference Budanitsky and Hirst2006; Lin Reference Lin1998). Syntax-based spaces represent a natural and needed evolution of semantic space models. Indeed, it has been observed by many that document-based and word-based spaces are often too simple to capture the structural complexity of texts. For the overall goal of understanding how meaning resides in texts, more complex linguistic features must to be taken into account: the study of syntax-based spaces represents a promising instantiation of this research line.

An open research challenge regards the attempt to better define how different distributional approaches can capture paradigmatic and syntagmatic properties. We know that different types of contexts emphasize either paradigmatic or syntagmatic properties, but their acquisition and use is also related to other factors: for example, the similarity measure available by a given semantic space. Symmetric similarity measures (e.g. distance metrics) are very important for acquiring and using paradigmatic classes, such as synonymy sets and taxonomic classes. On the contrary, antisymmetric relations and noncommutative functions (e.g. geometrical projections into subspaces) seem more important for directional inferences (e.g. lexical generalization or textual entailment).

Another promising area of research is the integration of different distributional models into a unifying one: Vector-based representations allow the unification of lexical representations emerging from different spaces, through the use of algebric composition. This represents a relevant advantage with respect to other representation formalisms, but requires more in depth exploration for its consequences in lexical semantics.

1.2 Distributional models and conceptual reuse

Distributional approaches to meaning have been successfully applied to a variety of problems related to language and beyond, thanks to their peculiar nature and properties.

A first important property is that distributional approaches allow to model different and equally rich context-sensitive semantic representations of a given language, inspired by a Wittgensteinian notion of language in use (Wittgenstein Reference Wittgenstein1953). By varying types of context, geometry, and similarity measures, these representations are still implemented as spaces, vectors, and embedded structures, such as graphs and functions. All of these express a contextual notion of meaning that can be successfully leveraged for the use and the maintenance of the corresponding lexical knowledge.

A second relevant property of distributional approaches, is that they embody a way to investigate lexical-semantic properties of a language, as a side-effect of the text mining processes that has been used to build the semantic space. This is beneficial for a truly empirical account of the language in use perspective on meaning.

Third, the set of spaces, vectors or analytical functions used as a representational device for lexical information provides a strong but unifying level of abstraction across heterogeneous cognitive representations and processes.

This third property is particularly interesting in the view of NLE. At the end of 1990s, NLE focused on the notion of algorithmic reuse as opposed to data reuse. The former was referring to the portability of individual algorithms and software tools. The latter focused on standardized annotation formalisms to favor the integration of labeled resources within different NLP processes and systems. The geometrical abstractions of distributional models support both notions. On the one side, they unify the representation formalisms: as real-valued vectors and matrices are very simple and semantically transparent data objects. On the other side, they allow a variety of algorithmic techniques, such as space transformations (e.g., or embeddings), to be easily ported across different tasks and linguistic levels. It can be noticed that semantic spaces enable even more sophisticated forms of reuse that we here call ‘conceptual’. A large set of heterogeneous cognitive phenomena can be in fact captured rather naturally by vector-based models (Gärdenfors Reference Gärdenfors2004).

Conceptual reuse suggests the application of geometrical paradigms as a bridge across different problems and data types. As opposed to heterogeneous paradigms specific to subtasks, geometry supports higher level abstractions, as well as a stronger unifying power. An example of a successful research based on a unified paradigm is the work in Globerson et al. (Reference Globerson, Chechik, Pereira and Tishby2007). The authors apply algorithms for geometrical embedding that handle different data types, such as images and texts, to search low-dimensional continuous representations for semantic data. A single common Euclidean space is thus derived, based on the co-occurrence statistics of the different data types. Once the embedding is performed, ‘. . . it induces a meaningful metric between objects of the same type. Such an approach may be used, for example, for embedding images based on accompanying text, and derive the semantic distance between images’. (Globerson et al. Reference Globerson, Chechik, Pereira and Tishby2007).

Several other examples of conceptual reuse can be found in the cognitive science literature. Computational approaches to cognitive models have been often advocating geometry and distributional theories as a representational paradigm able to integrate different levels of cognition (Gärdenfors Reference Gärdenfors2004; Widdows Reference Widdows2004). In particular, distributional techniques have been successfully applied to studies regarding metaphor detection and analysis (Kintsch Reference Kintsch2000), priming (Lowe and McDonald Reference Lowe and McDonald2000), discourse analysis(Landauer and Dumais Reference Landauer and Dumais1997), and neural activation analysis (Mitchell et al. Reference Mitchell, Shinkareva, Carlson, Vicente, Malva, Mason and Just2008). The underlying assumption is that the human cognitive ability of judging similarity among things and phenomena can be elegantly modeled or approximated by geometric spaces, which can provide powerful mathematical explanations of complex and effective notions of linguistic and conceptual similarity. ‘There is (. . .) no simple correspondence between cognitive functions and brain dynamics, and possibly different areas with different architecture may generate similar geometries and, hence, similar meaning content’ (Fenstad Reference Fenstad, Gerbrandy, Marx, de Rijke and Vossiuspers1999).Footnote ²

The above approaches are just additional examples with respect to the large body of literature on distributional methods in NLE. Distributional models have been successfully applied to several NLP tasks, such as word clustering (F. Pereira and Lee Reference Pereira, Tishby and Lee1993), harvesting thesauri (Lin Reference Lin1998), word sense disambiguation (Schutze Reference Schutze1998), acquisition of inference rules (Lin and Pantel Reference Lin and Pantel2001) and selectional preferences (Basili et al. Reference Basili, De Cao, Marocco and Pennacchiotti2007; Pantel et al. Reference Pantel, Bhagat, Coppola, Chklovski and Hovy2007), conceptual clustering (Pantel and Lin Reference Pantel and Lin2003), as well as the modeling of frame semantics phenomena (Pennacchiotti et al. Reference Pennacchiotti, De Cao, Basili, Croce and Roth2008) or question answering (Van der Plas Reference Van der Plas2008).

1.3 Distributional models and Natural Language Learning

The empirical nature of a distributional approach to lexical meaning enforces a strong connection with the entire field of machine learning. Contemporary research in machine learning emphasizes the role of geometrical spaces and distributional approaches in learning processes, with relevant applications to NLP tasks. Semantic spaces are relevant under the specific perspective referred as computational natural language learning. Two major specific trends are worth here to be emphasized. First, semantic spaces play a relevant role in the definition of linguistically principled kernel functions for supervised language learning. Second, they inherit a variety of algebraic techniques as extensions to basic vector space models, e.g. dimensionality reduction algorithms, able to effectively model unsupervised learning processes.

Sematic spaces and Kernels. In the mid-1990s, Statistical Learning Theory (Vapnik Reference Vapnik1995) shed some light on the relationships between empirical data analysis and learnability, through the mathematical characterization of the link between inductive problems and the classes of learnable functions to (Empirical Risk Minimization principle). This principle inspired the class of inductive algorithms known as Support Vector Machines (SVMs) (Vapnik Reference Vapnik1995). A key research area on SVMs is the study of kernel functions to model the learning problem (Haussler Reference Haussler1999; Cristianini and Shawe-Taylor Reference Cristianini and Shawe-Taylor2000). Kernels efficiently increase the expressiveness of a class of functions by also preserving desirable mathematical properties such as optimality, convergence and stability. In NLP, kernels have been explored for complex tasks, such as semantic role labeling and question answering, e.g. tree kernels (Collins and Duffy Reference Collins and Duffy2002; Moschitti Reference Moschitti2004; Moschitti, Pighin and Basili Reference Moschitti, Pighin and Basili2008) for syntactic similarity.

In short, kernels are defined as scalar products in metric spaces. The semantic spaces studied by distributional approaches induce scalar products that, by definition, embody meaningful linguistic information, as exhibited by the underlying corpora. These linguistically principled metric functions may act as lexical semantic kernel functions in a variety of supervised learning algorithms. The class of semantic kernels resulting from highly dimensional word spaces constitute a very effective integrated model of different linguistic features. Distributional types of linguistic kernels usually combine syntactic, distributional and prior semantic information, as for example explored in Bloehdorn et al. (Reference Bloehdorn, Basili, Cammisa and Moschitti2006), Bloehdorn and Moschitti (Reference Bloehdorn and Moschitti2007), and Ó Séaghdha and Copestake (Reference Ó and Copestake2008).

Algebraic transformations of Word Spaces for Unsupervised Learning. Vector space models and linear algebra have been largely applied for feature selection and dimensionality reduction. Here, (usually linear) transformations of the original vectors are proposed to induce more expressive and efficient feature spaces. Eigenvector analysis and Principal Component Analysis give rise to a number of variants, where feature spaces obtained by pure geometrical methods are used to capture more convenient representations for the original data. Geometric transformations for dimensionality reduction have been explored since 1990s (Landauer and Dumais Reference Landauer and Dumais1997; Hofmann Reference Hofmann1999; Tenenbaum, de Silva and Langford Reference Tenenbaum, de Silva and Langford2000). These aim at isolate the subset of relevant information implicit in a large semantic space, and representing this information in a new space, with minimal number of dimensions. Although several applications have already shown the huge impact of these methods in terms of improved accuracy and scalability, the implications on text classification and lexical acquisition tasks have not yet been fully explored.

To date, the first most relevant work on dimensionality reduction is Latent Semantic Indexing (Berry, Dumais and O'Brien Reference Berry, Dumais and O'Brien1995), in the framework of Information Retrieval. Latent Semantic Analysis (LSA) is an algorithm first presented by Furnas et al. (Reference Furnas, Deerwester, Dumais, Landauer, Harshman, Streeter and Lochbaum1988), and then by Landaur and Dumais (Reference Landauer and Dumais1997): Given an original space of documents, LSA finds the best subspace approximation of the original space, by minimizing the global reconstruction error, i.e. by projecting data along the directions of maximal variance.

More recently, new researches have emphasized the role of local information in the original space, as opposed to global ones, in those cases in which euclidean metrics have only local validity. Typical approaches try to exploit the geometry of the underlying manifold in which the data are described, in form of a variety of manifold-based learning algorithms (Roweis and Saul Reference Roweis and Saul2000; Tenenbaum et al. Reference Tenenbaum, de Silva and Langford2000; Saul and Roweis Reference Saul and Roweis2003). Isomap (Tenenbaum et al. Reference Tenenbaum, de Silva and Langford2000) was originally proposed as a generalization of multidimensional scaling, where symmetric adjacency graphs (based on criteria such as symmetric nearest neighborhoods) are used as opposed to Euclidean distances in the high-dimensional space. The geodesic distance on the manifold allows to unfold the original space and improve the generalization in classification and clustering. New approaches have been then proposed, such as Locally Linear Embedding (LLE) (Roweis and Saul Reference Roweis and Saul2000) and Locality Preserving Projection (He and Niyogi Reference He and Niyogi2003). In LLE, each high-dimensional input is mapped into a low-dimensional output representing global internal coordinates on a locally Euclidean manifold. All these results are characterized by some geometrical assumptions, such as the clustering hypothesis, and are used to promote rich topologies out from the source data distribution. The transformations of the original representation space determines novel metrics within the original feature space.

In synthesis, this class of methods allow to learn the metrics from the data themselves, and are particularly attractive for linguistic tasks. Successful examples of these approaches are latent semantic kernels (Cristianini, Shawe-Taylor and Lodhi Reference Cristianini, Shawe-Taylor and Lodhi2002) and Manifold Regularization for semisupervised learning (Belkin, Niyogi and Sindhwani Reference Belkin, Niyogi and Sindhwani2006; Sindhwani, Belkin and Niyogi Reference Sindhwani, Belkin, Niyogi, Chapelle, and Zien2006).

1.4 Perspectives for distributional approaches

Metric spaces are used in many natural sciences, such as biology and physics, to model domains, as typed feature structures in Pollard and Sag (Reference Pollard and Sag1994). The body of work we described so far, shows that metric spaces and geometry can be successfully used as modeling domain for language semantics. Reasoning over mathematical abstractions such as vectors and probability (i.e. density) functions can be reasonably expected to say a lot about intrinsic properties of natural language semantics. This perspective is nowadays even more critical, as for the urgency of understanding the interplays between the social and linguistic dimensions of text semantics in the Web 2.0 era. Geometrical models are here useful along two directions.

First, semantic spaces provide a framework where linguistic phenomena can be studied through a mathematics that helps to account for the discrete but nonlinear nature of linguistic phenomena (Manifold Learning). The manifold assumption states that a usually low-dimensional and possibly nonlinear manifold accounts for the distributions of observable data; and that categorical knowledge can be seen as a smooth function over these manifolds. The application of this perspective in NLE studies consists in the search of linguistically motivated manifolds that correspond to useful generalizations. As a consequence, weakly supervised (or even unsupervised) learning models over these manifolds can be designed. A large family of semisupervised learning methods have been recently proposed to exploit the distributions of unlabeled data, as a way to maximize the generalization accuracy by learning from small sets of labeled data (Sindhwani et al. Reference Sindhwani, Belkin, Niyogi, Chapelle, and Zien2006). In such a way, unlabeled examples are used to derive information about the underlying implicit manifold, that is very likely non-Euclidean. They maximize the predictive accuracy of known supervised algorithms (e.g. SVMs) by exploiting the geometry of the underlying manifold, i.e. the distributions intrinsic to the linguistic data sample. As the amount of unlabeled data is usually very large in language learning problems, this view is very attractive in NLE although not yet fully explored. Moreover, this perspective is even more appealing, as the information about the manifold can in principle be also gathered from a priori linguistic knowledge. In facts, nothing prevents to impose a priori restrictions on the unlabeled data, so that their distribution is forced to the specific interpretation inspired by a resource, e.g. a semantic dictionary. In this case, the manifold is artificially superimposed, but the mathematics of the above manifold-based semisupervised learning is fully preserved. The result is an elegant way of integrating lexical knowledge as a set of a priori constraints over an embedding into the manifold in which the learning takes place. A noticeable example applied to sentiment analysis is recently discussed in Sindhwani and Melville (Reference Sindhwani and Melville2008). Another work where dimensionality reduction methods are applied to elegantly integrate a priori information is (Weinberger and Chapelle Reference Weinberger, Chapelle, Koller, Schuurmans, Bengio and Bottou2008), where a topic taxonomy is embedded within a document representation space, and accordingly an effective semisupervised document classification method is obtained.

Vector Spaces are also important models for analytical descriptions of uncertain systems, and in particular incomplete systems. The physiological and psychological aspects of language have been often recognized as out of the scope of systems characterized by a complete knowledge. In analogy with quantum physics, incompleteness is rather standard for the knowledge available to many linguistic inferences. The logic developed in this frameworks, i.e. quantum logic (Birkhoff and von Neumann Reference Birkhoff and von Neumann1936), is a model of inference with respect of a variety of uncertain conditions. The crucial properties for such logics are, among others, graded notions of similarity and noncommutativity of several operators (Varadarajan Reference Varadarajan1985). In (van Rijsbergen Reference van Rijsbergen2004) well-known measures, as projections in subspaces, are discussed as significant noncommutative operators. While measurement in classical mechanics is always commutative, quantum mechanics allows naturally for its noncommutativity. Composition of projections as well as tensor products, are both useful operators for directional inference in quantum mechanics. The impact of these possibilities on language processing models is clear. Language technologies can look at semantic spaces with a strong interest not only on symmetrical relations (e.g. as the notion semantic similarity measures traditionally modeled for sense discrimination or disambiguation tasks, Schutze Reference Schutze1998), but also for complex directional linguistic inferences, e.g. textual entailment.