2.1 The distributional event graph

The Distributional Event Graph represents the event knowledge stored in long-term memory with information extracted from parsed corpora. We assume a very broad notion of event, as an n-ary relation between entities. Accordingly, an event can be a complex situation involving multiple participants, such as The student reads a book in the library, but also the association between an entity and a property expressed by the noun phrase heavy book. This notion of event corresponds to what psychologist call situation knowledge or thematic associations (Binder Reference Binder2016). As McRae and Matsuki (Reference McRae and Matsuki2009) argue, gek is acquired from both sensorimotor experience (e.g. watching or playing football matches) and linguistic experience (e.g. reading about football matches). deg can thus be regarded as a model of the gek derived from the linguistic input.

Events are extracted from parsed sentences, using syntactic relations as an approximation of deeper semantic roles (e.g. the subject relation for the agent and the direct object relation for the patient). In the present paper, we use dependency parses, as it is customary in distributional semantics, but nothing in SDM hinges on the choice of the syntactic representation. Given a verb or a noun head, all its syntactic dependents are grouped together.Footnote ^c More schematic events are also generated by abstracting from one or more event participants for every recorded instance. Since we expect each participant to be able to trigger the event and consequently any of the other participants, a relation can be created and added to the graph from every subset of each group extracted from a sentence (cf. Figure 1).

Figure 1:

Reduced version of the parsing for the sentence The student is reading the book about Shakespeare in the university library. Three events are identified, each represented with a dotted box.

The resulting deg structure is a weighted hypergraph, as it contains weighted relations holding between nodes pairs, and a labelled multigraph, since the edges are labelled in order to represent specific syntactic relations. The weights σ are derived from co-occurrence statistics and measure the association strengths between event nodes. They are intended as salience scores that identify the most prototypical events associated with an entity (e.g. the typical actions performed by a student). Crucially, the graph nodes are represented as word embeddings. Thus, given a lexical cue w, the information in deg can be activated along two dimensions during processing (cf. Table 1):

Table 1:

The five nearest paradigmatic and syntagmatic neighbours for the lexical item book, extracted from deg

1. (1) by retrieving the most similar nodes to w (the paradigmatic neighbours), on the basis of their cosine similarity between their vectors and the vector of w;

2. (2) by retrieving the closest associates of w (the syntagmatic neighbours), using the edge weights.

Figure 2 shows a toy example of deg. The little boxes with circles in them represent the embedding associated with each node. Edges are labelled with syntactic relations (as a surface approximation of event roles) and weighted with salience scores σ. Each event is a set of co-indexed edges. For example, e ₂ corresponds to the event of students reading books in libraries, while e ₁ represents a schematic event of students performing some generic action on books (e.g. reading, consulting and studying).

Figure 2:

Toy sample of deg showing several instances of events, each represented by a sequence of co-indexed e. The σ are the event salience weights.

2.2 The meaning composition function

We assume that during sentence comprehension lexical items activate fragments of event knowledge stored in deg (like in Elman’s Words-as-Cues model), which are then dynamically integrated in a semantic representation sr. This is a formal structure directly inspired by DRT and consisting of three different yet interacting information tiers:

• (1) universe (U) – this tier, which we do not discuss further in the present paper, includes the entities mentioned in the sentence (corresponding to the discourse referents in DRT). They are typically introduced by noun phrases and provide the targets of anaphoric links.

• (2) linguistic conditions (lc) – a context-independent tier of meaning that accumulates the embeddings associated with the lexical items. This corresponds to the conditions that in DRT content words add to the discourse referents. The crucial difference is that now such conditions are embeddings.

• (3) active context (ac) – similarly to the notion of articulated context in Kamp (Reference Kamp2016), this component consists of several types of contextual information available during sentence processing or activated by lexical items (e.g. information from the current communication setting and general world knowledge). More specifically, we assume that ac contains the embeddings activated from deg by the single lexemes (or by other contextual elements) and integrated into a semantically coherent structure contributing to the sentence interpretation.

Figure 3 shows an example of sr built from the sentence The student drinks the coffee (ignoring the specific contribution of determiners and tense). The universe U contains the discourse referents introduced by the noun phrases, while lc includes the embeddings of the lexical items in the sentence, each linked to the relevant referent (e.g. $$\overrightarrow {student}$$ : u means that the embedding introduced by student is linked to the discourse referent u). ac consists of the embeddings activated from deg and ranked by their salience with respect to the current content in the sr. The elements in ac are grouped by their syntactic relation in deg, which again we regard here just as a surface approximation of their semantic role (e.g. the items listed under ‘obl:loc’ are a set of possible locations of the event expressed by the sentence). ac makes it possible to enrich the semantic content of the sentence with contextual information, predict other elements of the event and generate expectations about incoming input. For instance, given the ac in Figure 3, we can predict that the student is most likely to be drinking a coffee at the cafeteria and that he/she is drinking it for breakfast or in the morning. The ranking of each element in ac depends on two factors: (i) its degree of activation by the lexical items and (ii) its overall coherence with respect to the information already available in the ac.

Figure 3:

Sample sr for the sentence The student drinks the coffee. The sentence activates typical locations and times in which the event could take place.

A crucial feature of each sr is that lc and ac are also represented with vectors that are incrementally updated with the information activated by lexical items. Let sr_i-1 be the semantic representation built for the linguistic input w ₁, …, w _i−1. When we process a new pair 〈w_i, r_i 〉 with a lexeme w_i and syntactic role r_i:

1. 1. lc in sr_i−1 is updated with the embedding $$\overrightarrow {w_i }$$;

2. 2. ac in sr_i−1 is updated with the embeddings of the syntagmatic neighbours of w_i extracted from DEG.

Figures 4 and 5 exemplify the update of the sr for the subject The student with the information is activated by the verb drink. The update process is defined as follows:

1. (1) lc is represented with the vector $$\overrightarrow {LC}$$ obtained from the linear combination of the embeddings of the words contained in the sentence. Therefore, when $$\langle w_i ,r_i \rangle$$ is processed, the embedding $$\overrightarrow {w_i }$$ is simply added to $$\overrightarrow {LC}$$;Footnote ^d

2. (2) for each syntactic role r_i, ac contains a set of ranked lists (one for each processed pair) of embeddings corresponding to the most likely words expected to fill that role. For instance, the ac for the fragment The student in Figure 4 contains a list of the embeddings of the most expected direct objects associated with student, a list of the embeddings of the most expected locations, etc. Each list of expected role fillers is itself represented with the weighted centroid vector (e.g. $$\overrightarrow {dobj}$$) of their k most prominent items (with k a model hyperparameter). For instance, setting k = 2, the $$\overrightarrow {dobj}$$ centroid in the ac in Figure 4 is built just from $$\overrightarrow {book}$$ and $$\overrightarrow {research}$$; less salient elements (the gray areas in Figures 3–5) are kept in the list of likely direct objects, but at this stage do not contribute to the centroid representing the expected fillers for that role. ac is then updated with the deg fragment activated by the new lexeme w_i (e.g. the verb drink):

   • The event knowledge activated by w_i for a given role r_i is ranked according to cosine similarity with the vector $$\overrightarrow {r_i }$$available in ac: in our example, the direct objects activated by the verb drink (e.g. $$\overrightarrow {beer}$$ and $$\overrightarrow {coffee}$$) are ranked according to their cosine similarity to the $$\overrightarrow {dobj}$$ vector of the ac.

   • The ranking process works also in the opposite direction: the newly retrieved information is used to update the centroids in ac. For example, the direct objects activated by the verb drink are aggregated into centroids and the corresponding weighted lists in ac are re-ranked according to the cosine similarity with the new centroids, in order to maximize the semantic coherence of the representation. At this point, $$\overrightarrow {book}$$ and $$\overrightarrow {research}$$, which are not as salient as $$\overrightarrow {coffee}$$ and $$\overrightarrow {beer}$$ in the drinking context, are downgraded in the ranked list and are therefore less likely to become part of the $$\overrightarrow {dobj}$$ centroid at the next step.

   The newly retrieved information is now added to the ac: as shown in Figure 5, once the pair 〈drink, root〉 has been fully processed, the ac contains two ranked lists for the dobj role and two ranked lists for the obl:loc role, the top k elements of each list will be part of the centroid for their relation in the next step. Finally, the whole ac is represented with the centroid vector $$\overrightarrow {AC}$$ built out of the role vectors $$\overrightarrow {r_1 } , \ldots ,\overrightarrow {r_n }$$ available in ac. The vector $$\overrightarrow {AC}$$ encodes the integrated event knowledge activated by the linguistic input.

Figure 4:

On the left, the sr for The student. On the right, the embedding and deg portion activated by the verb drink.

Figure 5:

The original semantic representation sr for The student … is updated with the information activated by the verb, producing the sr for The student drinks … The new event knowledge is re-ranked with respect to the previous content of ac.

As an example of gek re-ranking, assume that after processing the subject noun phrase The student, the ac of the corresponding sr predicts that the most expected verbs are read, study, drink, etc., the most expected associated direct objects are book, research, beer, etc., and the most expected locations are library, cafeteria, university, etc. (Figure 4). When the main verb drink is processed, the corresponding role list is removed by the ac, because that syntactic slot is now overtly filled by this lexeme, whose embedding is then added to the lc. The verb drink cues its own event knowledge, for instance, that the most typical objects of drinking are tea, coffee, beer, etc., and the most typical locations are cafeteria, pub, bar, etc. The information cued by drink is re-ranked to promote those items that are most compatible and coherent with the current content of ac (i.e. direct objects and locations that are likely to interact with students). Analogously, the information in the ac is re-ranked to make it more compatible with the gek cued by drink (e.g. the salience of book and research gets decreased, because they are not similar to the typical direct objects and locations of drink). The output of the sr update is shown in Figure 5, whose ac now contains the gek associated with an event of drinking by a student.

A crucial feature of sr is that it is a much richer representation than the bare linguistic input: the overtly realized arguments in fact activate a broader array of roles than the ones actually appearing in the sentence. As an example of how these unexpressed arguments contribute to the semantic representation of the event, consider a situation in which three different sentences are represented by means of ac, namely The student writes the thesis, The headmaster writes the review and The teacher writes the assignment. Although teacher could be judged as closer to headmaster than to student, and thesis as closer to assignment than to review, taking into account also the typical locations (e.g. a library for the first two sentences, a classroom for the last one) and writing supports (e.g. a laptop in the first two cases, a blackboard in the last one) would lead to the first two events being judged as the most similar ones.

In the case of unexpected continuations, the ac will be updated with the new information, though in this case the re-ranking process would probably not change the gek prominence. Consider the case of an input fragment like The student plows…: student activates event knowledge as it is shown in Figure 3, but the verb does not belong to the set of expected events given student. The verb triggers different direct objects from those already in the ac (e.g. typical objects of plow such as furrow and field). Since the similarity of their centroid with the elements of the direct object list in the ac will be very low, the relative ordering of the ranked list will roughly stay the same, and direct objects pertaining to the plowing situation will coexist with direct objects triggered by student. Depending on the continuation of the sentence, then, the elements triggered by plow might gain centrality in the representation or remain peripheral.

It is worth noting that the incremental process of the sr update is consistent with the main principles of formal dynamic semantics frameworks like DRT. As we said above, dynamics semantics assumes the meaning of an expression to be a context-change potential that affects the interpretation of the following expressions. Similarly, in our distributional model of sentence representation the ac in sr_i−1 affects the interpretation of the incoming input w_i, via the gek re-ranking process.Footnote ^e

3.1 Data sets and tasks

Our goal is to test SDM in compositionality-related tasks, with a particular focus on the contribution of event knowledge. For the present study, we selected two different data sets: the development set of the RELPRON data set (Rimell et al. Reference Rimell, Maillard, Polajnar and Clark2016)Footnote ^f and the DTFit data set (Vassallo et al. Reference Vassallo, Chersoni, Santus, Lenci and Blache2018).

RELPRON consists of 518 target–property pairs, where the target is a noun labelled with a syntactic function (either subject or direct object) and the property is a subject or object relative clause providing the definition of the target (Figure 6). Given a model, we produce a compositional representation for each of the properties. In each definition, the verb, the head noun and the argument are composed to obtain a representation of the property. Following the original evaluation in Rimell et al. (Reference Rimell, Maillard, Polajnar and Clark2016), we tested six different combinations for each composition model: the verb only, the argument only, the head noun and the verb, the head noun and the argument, the verb and the argument and all three of them. For each target, the 518 composed vectors are ranked according to their cosine similarity to the target. Like Rimell et al. (Reference Rimell, Maillard, Polajnar and Clark2016), we use mean average precision (henceforth MAP) to evaluate our models on RELPRON. Formally, MAP is defined as

(1) $$MAP = \frac{1} {N}\sum\limits_{i = 1}^N AP(t_i ) $$

Figure 6:

Image from Rimell et al. (Reference Rimell, Maillard, Polajnar and Clark2016), showing the terminology for terms and properties in RELPRON: subject relative clause top, object relative clause bottom.

where N is the number of terms in RELPRON, and AP(t) is the average precision for term t, defined as

(2) $$ AP(t) = \frac{1} {{P_t }}\sum\limits_{k = 1}^M Prec(k) \times rel(k) $$

where P_t is the number of correct properties for term t in the data set, M is the total number of properties in the data set, Prec(k) is the precision at rank k and rel(k) is a function equal to one if the property at rank k is a correct property for t, and zero otherwise. Intuitively, AP(t) will be if, for the term t, all the correct properties associated to the term are ranked in the top positions, and the value becomes lower when the correct items are ranked farther from the head of the list.

Our second evaluation data set, DTFit, has been introduced with the goal of building a new gold standard for the thematic fit estimation task (Vassallo et al. Reference Vassallo, Chersoni, Santus, Lenci and Blache2018). Thematic fit is a psycholinguistic notion similar to selectional preferences, the main difference being that the latter involve the satisfaction of constraints on discrete semantic features of the arguments, while thematic fit is a continuous value expressing the degree of compatibility between an argument and a semantic role (McRae et al. Reference McRae, Spivey-Knowlton and Tanenhaus1998). Distributional models for thematic fit estimation have been proposed by several authors (Erk Reference Erk2007; Baroni and Lenci Reference Baroni and Lenci2010; Erk et al. Reference Erk, Padó and Padó2010; Lenci Reference Lenci2011; Sayeed et al. Reference Greenberg, Sayeed and Demberg2015; Greenberg et al. Reference Greenberg, Sayeed and Demberg2015; Santus et al. Reference Santus, Chersoni, Lenci and Blache2017; Tilk et al. Reference Tilk, Demberg, Sayeed, Klakow and Thater2016; Hong et al. Reference Hong, Sayeed and Demberg2018). While thematic fit data sets typically include human-elicited typicality scores for argument–filler pairs taken in isolation, DTFit includes tuples of arguments of different length, so that the typicality value of an argument depends on its interaction with the other arguments in the tuple. This makes it possible to model the dynamic aspect of argument typicality, since the expectations on an argument are dynamically updated as the other roles in the sentence are filled. The argument combinations in DTFit describe events associated with crowdsourced scores ranging from 1 (very atypical) to 7 (very typical). The data set items are grouped into typical and atypical pairs that differ only for one argument, and divided into three subsets:

• 795 triplets, each differing only for the Patient role:

  • - sergeant_N assign_V mission_N (typical)

  • - sergeant_N assign_V homework_N (atypical)

• 300 quadruples, each differing only for the Location role:

  • - policeman_N check_V bag_N airport_N (typical)

  • - policeman_N check_V bag_N kitchen_N (atypical)

• 200 quadruples, each differing only for the Instrument role:

  • - painter_N decorate_V wall_N brush_N (typical)

  • - painter_N decorate_V wall_N scalpel_N (atypical)

However, the Instrument subset of DTFit was excluded from our current evaluation. After applying the threshold of 5 for storing events in the deg (cf. Section 3.2.3), we found that the SDM coverage on this subset was too low.

For each tuple in the DTFit data set, the task for our models is to predict the upcoming argument on the basis of the previous ones. Given a model, we build a compositional vector representation for each data set item by excluding the last argument in the tuple, and then we measured the cosine similarity between the resulting vector and the argument vector. Models are evaluated in terms of the Spearman’s correlation between the similarity scores and the human ratings.

As suggested by the experimental results of Bicknell et al. (Reference Bicknell, Elman, Hare, McRae and Kutas2010) and Matsuki et al. (Reference Matsuki, Chow, Hare, Elman, Scheepers and McRae2011), the typicality of the described events has important processing consequences: atypical events lead to longer reading times and stronger N400 components, while typical ones are easier to process thanks to the contribution of gek. Thus, the task of modelling typicality judgements can be seen as closely related to modelling semantic processing complexity.

3.2 Models settings

In this study, we compare the performance of SDM with three baselines. The simple additive model formulated in Mitchell and Lapata (Reference Mitchell and Lapata2010), a smoothed additive model and a multilayer Long-Short-Term-Memory (LSTM) neural language model trained against one-hot targets (Zaremba et al. Reference Zaremba, Sutskever and Vinyals2015).

The additive models (Mitchell and Lapata Reference Mitchell and Lapata2010) have been evaluated on different types of word embeddings. We compared their performances with SDM.Footnote ^g Despite their simplicity, previous evaluation studies on several benchmarks showed that such models can be difficult to beat, even for sophisticated compositionality frameworks (Rimell et al. Reference Rimell, Maillard, Polajnar and Clark2016; Arora et al. Reference Arora, Liang and Ma2017; Tian et al. Reference Tian, Okazaki and Inui2017).

The embeddings we used in our tests are the word2vec models by Mikolov et al. (Reference Mikolov, Sutskever, Chen, Corrado and Dean2013), that is the Skip-Gram with negative sampling (SG) and the continuous-bag-of-words (CBOW), and the C-Phrase model by Pham et al. (Reference Pham, Kruszewski, Lazaridou and Baroni2015). The latter model incorporates information about syntactic constituents, as the principles of the model training are (i) to group the words together according to the syntactic structure of the sentences and (ii) to optimize simultaneously the context predictions at different levels of the syntactic hierarchy (e.g. given the training sentence A sad dog is howling in the park, the context prediction will be optimized for dog, a dog and a sad dog, that is for all the words that form a syntactic constituent). The performance of C-Phrase is particularly useful to assess the benefits of using vectors that encode directly structural/syntactic information.

We used the same corpora both for training the embeddings and for extracting the syntactic relations for deg. The training data come from the concatenation of three dependency-parsed corpora: the BNC (Leech and Smith Reference Leech2000), the Ukwac (Baroni et al. Reference Baroni, Bernardini, Ferraresi and Zanchetta2009) and a 2018 dump of the English Wikipedia, for a combined size of approximately 4 billion tokens. The corpora were parsed with Stanford CoreNLP (Manning et al. Reference Manning, Surdeanu, Bauer, Finkel, Bethard and McClosky2014). The hyperparameters of the embeddings were the following for all models: 400 dimensions, a context window of size 10, 10 negative samples and 100 as the minimum word frequency.Footnote ^h

3.2.1 Simple additive models

Our additive models, corresponding to an sr consisting of the $$\overrightarrow {LC}$$ component only, represent the meaning of a sentence sent by summing the embeddings of its words:

(3) $$ \overrightarrow {sent} = \sum\limits_{w \in sent} \vec w$$

The similarity with the targets is measured with the cosine between the target vector and the sentence vector.

3.2.2 Smoothed additive models

These models are a smoothed version of the additive baseline, in which the final representation is simply the sum of the vectors of the words in the sentence, plus the top k = 5 nearest neighbour of each word in the sentence.Footnote ⁱ Therefore, the meaning of a sentence sent is obtained by

(4) $$ \overrightarrow {sent} = \sum\limits_{w \in sent} \left( {\vec w + \sum\limits_{x \in N_5 (w)} \vec x} \right) $$

where N_k(w) is the set of the k nearest neighbours of w. Compared to the gek models, the smoothed additive baseline modifies the sentence vector by adding the vectors of related words. Thus, it represents a useful comparison term for understanding the actual added value of the structural aspects of SDM.Footnote ^j

3.2.3 The structured distributional models

The SDM introduced in Section 2 consists of a full sr including the linguistic conditions vector $$\overrightarrow {LC}$$ and the event knowledge vector $$\overrightarrow {AC}$$. In this section, we detail the hyperparameter setting for the actual implementation of the model.

Distributional event graph

We included in the graph only events with a minimum frequency of 5 in the training corpora. The edges of the graph were weighted with Smoothed Local Mutual Information (LMI). Given a triple composed by the words w ₁ and w ₂, and a syntactic relation s linking them, we computed its weight by using a smoothed version of the Local Mutual Information (Evert Reference Evert2004):

(5) $$ LMI_\alpha (w_1 ,w_2 ,s) = f(w_1 ,w_2 ,s)*log(\frac{{P(w_1 ,w_2 ,s)}} {{P(w_1 )*P_\alpha (w_2 )*P(s)}}) $$

where the smoothed probabilities are defined as follows:

(6) $$ P_\alpha (x) = \frac{{f(x)^\alpha }} {{\sum\limits_x f(x)^\alpha }} $$

This type of smoothing, with α = 0.75, was chosen to mitigate the bias of Mutual Information (MI) statistical association measures towards rare events (Levy et al. Reference Levy, Goldberg and Dagan2015). While this formula only involves pairs (as only pairs were employed in the experiments), it is easily extensible to more complex tuples of elements.

Re-ranking settings

For each word in the data set items, the top 50 associated words were retrieved from deg. Both for the re-ranking phase and for the construction of the final representation, the event knowledge vectors (i.e. the role vectors $$\overrightarrow r$$ and the ac vector $$\overrightarrow {AC}$$) are built from the top 20 elements of each weighted list. As detailed in Section 2.2, the ranking process in SDM can be performed in the forward direction and in the backward direction at the same time (i.e. the ac can be used to re-rank newly retrieved information and vice versa, respectively), but for simplicity we only implemented the forward ranking.

Scoring

As in SDM the similarity computations with the target words involves two separate vectors, we combined the similarity scores with addition. Thus, given a target word in a sentence sent, the score for SDM will be computed as

(7) $$ score(target,sent) = cos(\overrightarrow {target} ,\overrightarrow {LC} (sent)) + cos(\overrightarrow {target} ,\overrightarrow {AC} (sent))$$

In all settings, we assume the model to be aware of the syntactic parse of the test items. In DTFit, word order fully determines the syntactic constituents, as the sentences are always in the subject verb object [location-obl|instrument-obl] order. In RELPRON, on the other hand, the item contains information about the relation that is being tested: in the subject relative clauses, the properties always show the verb followed by the argument (e.g. telescope: device that detects planets), while in the object relative clauses the properties always present the opposite situation (e.g. telescope: device that observatory has). In the present experiments, we did not use the predictions on non-expressed arguments to compute $$\overrightarrow {AC}$$, and we restricted the evaluation to the representation of the target argument. For example, in the DTFit Patients set, $$\overrightarrow {AC} (sent)$$ only contains the $$\overrightarrow {dobj}$$centroid.

4.1 RELPRON

Given the targets and the composed vectors of all the definitions in RELPRON, we assessed the cosine similarity of each pair and computed the MAP scores shown in Table 2. First of all, the Skip-Gram-based models always turn out to be the best performing ones, with rare exceptions, closely followed by the C-Phrase ones. The scores of the additive models are slightly inferior, but very close to those reported by Rimell et al. (Reference Rimell, Maillard, Polajnar and Clark2016), while the LSTM model lags behind vector addition, improving only when the parameter N is increased. Results seem to confirm the original findings: even with very complex models (in that case, the Lexical Function Model by Paperno et al. Reference Paperno, Pham and Baroni2014), it is difficult to outperform simple vector addition in compositionality tasks.

Table 2:

Results for the vector addition baseline, smoothed vector addition baseline, LSTM and the SDM on the RELPRON development set (MAP scores). Rows refer to the different word combinations tested in Rimell et al. 2016 (R&al.)

Interestingly, SDM shows a constant improvement over the simple vector addition equivalents (Table 2), with the only exception of the composition of the verb and the argument. All the results for the headNoun+verb+arg composition are, to the best of our knowledge, the best scores reported so far on the data set. Unfortunately, given the relatively small size of RELPRON, the improvement of the gek models fails to reach significance (p > 0.1 for all comparisons between a basic additive model and its respective augmentation with deg, p values computed with the Wilcoxon rank-sum test). Compared to SDM, the smoothed vector addition baseline seems to be way less consistent (Table 2): for some combinations and for some vector types, adding the nearest neighbours is detrimental. We take these results as supporting the added value of the structured event knowledge and the sr update process in SDM, over the simple enrichment of vector addition with nearest neighbours. Finally, we can notice that the Skip-Gram vectors have again an edge over the competitors, even over the syntactically informed C-Phrase vectors.

Table 3:

Results for the vector addition baseline, smoothed vector addition baseline, LSTM and the SDM on the Patients and Locations subsets of DTFit. The scores are expressed in terms of Spearman’s correlation with the gold standard ratings. The LSTM scores refer to the best configuration, with N = 10 and vectors of size 400. The statistical significance of the improvements over the additive baseline is reported as follows: *p < 0.05; **p < 0.01 (p values computed with Fisher’s r-to-z transformation, one-tailed test). ns = non-significant correlation

4.2 DTFit

At a first glance, the results on DTFit follow a similar pattern (Table 3): the three embedding types perform similarly, although in this case the CBOW vectors perform much worse than the others in the Patients data set. LSTM also largely lags behind all the additive models, showing that thematic fit modelling is not a trivial task for language models, and that more complex neural architectures are required in order to obtain state-of-the-art results (Tilk et al. Reference Tilk, Demberg, Sayeed, Klakow and Thater2016).Footnote ^k

The results for SDM again show that including the deg information leads to improvements in the performances (Table 3). While on the Locations the difference is only marginal, also due to the smaller number of test items, two models out of three showed significantly higher correlations than their respective additive baselines. The increase is particularly noticeable for the CBOW vectors that, in their augmented version, manage to fill the gap with the other models and to achieve a competitive performance. However, it should also be noticed that there is a striking difference between the two subsets of DTFit: while on patients the advantage of the gek models on both the baselines is clear, on locations the results are almost indistinguishable from those of the smoothed additive baseline, which simply adds the nearest neighbours to the vectors of the words in the sentence. This complies with previous studies on thematic fit modelling with dependency-based distributional models (Sayeed et al. 2015; Santus et al. Reference Chersoni, Santus, Blache and Lenci2017). Because of the ambiguous nature of the prepositions used to identify potential locations, the role vectors used by SDM can be very noisy. Moreover, since most locative complements are optional adjuncts, it is likely that the event knowledge extracted from corpora contain a much smaller number of locations. Therefore, the structural information about locations in deg is probably less reliable and does not provide any clear advantage compared to additive models.

Concerning the comparison between the different types of embeddings, Skip-Gram still retains an advantage over C-Phrase in its basic version, while it is outperformed when the latter vectors are used in SDM. However, the differences are clearly minimal, suggesting that the structured knowledge encoded in the C-Phrase embeddings is not a plus for the thematic fit task. Concerning this point, it must be mentioned that most of the current models for thematic fit estimation rely on vectors relying either on syntactic information (Baroni and Lenci Reference Baroni and Lenci2010; Greenberg et al. Reference Greenberg, Sayeed and Demberg2015; Santus et al. Reference Santus, Chersoni, Lenci and Blache2017; Chersoni et al. Reference Chersoni, Lenci and Blache2017) or semantic roles (Sayeed et al. Reference Sayeed, Demberg and Shkadzko2015; Tilk et al. Reference Tilk, Demberg, Sayeed, Klakow and Thater2016). On the other hand, our results comply with studies like Lapesa and Evert (Reference Lapesa and Evert2017), who reported comparable performance for bag-of-words and dependency-based models on several semantic modelling tasks, thus questioning whether the injection of linguistic structure in the word vectors is actually worth its processing cost. However, this is the first time that such a comparison is carried out on the basis of the DTFit data set, while previous studies proposed slightly different versions of the task and evaluated their systems on different benchmarks.Footnote ^l A more extensive and in-depth study is required in order to formulate more conclusive arguments on this issue.

Table 4:

Spearman’s correlations of the vector addition baseline and the SDM based on Skip-Gram on the DTFit patients subset

Another constant finding of previous studies on thematic fit modelling was that high-dimensional, count-based vector representations perform generally better than dense word embeddings, to the point that Sayeed et al. (Reference Sayeed, Greenberg and Demberg2016) stressed the sensitivity of this task to linguistic detail and to the interpretability of the vector space. Therefore, we tested whether vector dimensionality had an impact on task performance (Table 4). Although the observed differences are generally small, we noticed that higher-dimensional vectors are generally better in the DTFit evaluation and, in one case, the differences reach a marginal significance (i.e. the difference between the 100-dimensional and the 400-dimensional basic Skip-Gram model is marginally significant at p < 0.1). This point will also deserve future investigation, but it seems plausible that for this task embeddings benefit from higher dimensionality for encoding more information, as it has been suggested by Sayeed and colleagues. However, these advantages do not seem to be related to the injection of linguistic structure directly in the embeddings (i.e. not to the direct use of syntactic contexts for training the vectors), as bag-of-words models perform similarly to – if not better than – a syntactic-based model like C-Phrase. We leave to future research a systematic comparison with sparse count-based models to assess whether interpretable dimensions are advantageous for modelling context-sensitive thematic fit.

4.3 Error analysis

One of our basic assumptions about gek is that semantic memory stores representations of typical events and their participants. Therefore, we expect that integrating gek into our models might lead to an improvement especially on the typical items of the DTFit data set. A quick test with the correlations revealed that this is actually the case (Table 5): all models showed increased Spearman’s correlations on the tuples in the typical condition (and in the larger Patients subset of DTFit, the increase is significant at p < 0.05 for the CBOW model), while they remain unchanged or even decrease for the tuples in the atypical conditions. Notice that this is true only for SDM, which is enriched with gek. On the other hand, the simple addition of the nearest neighbours never leads to improvements, as proved by the low correlation scores of the smoothed additive baseline. As new and larger data sets for compositionality tasks are currently under construction (Vassallo et al. Reference Vassallo, Chersoni, Santus, Lenci and Blache2018), it will be interesting to assess the consistency of these results.

Table 5:

Comparison of the performance of the vector addition baseline, smoothed vector addition baseline and the SDM on the typical items of DTFit Patients (Spearman’s correlations). Δ reports the SDM improvements over the basic additive models. Significance is noted with the following notation: *p < 0.05

Turning to the RELPRON data set, we noticed that the difference between subject and object relative clauses is particularly relevant for SDM, which generally shows better performances on the latter. Table 6 summarizes the scores component on the two subsets. While relying on syntactic dependencies, SDM also processes properties in linear order: the verb+arg model, therefore, works differently when applied to subject clauses than to object clauses. In the subject case, in fact, the verb is found first, and then its expectations are used to re-rank the object ones. In the object case, on the other hand, things proceed the opposite way: at first the subject is found, and then its expectations are used to re-rank the verb ones. Therefore, the event knowledge triggered by the verb seems not only less informative than the one triggered by the argument, but it is often detrimental to the composition process.

Table 6:

Comparison of the SDM performance (MAP) on the subject and object relative clauses in RELPRON