4.1 Approaches based on explicit word vectors

The approaches falling into this group use explicit word vectors to characterize the state-of-affairs of word senses. In an explicit word vector, each cell in the vector corresponds to association strength between the target word and a context word, namely a word occurring in the context of the target word. The context word can be identified in different ways. It can be a collocate of the target word. For instance, Tang et al. (Reference Tang, Qu and Chen2016) represent a sense instance with a tuple c = (w _t,w _m), in which w _t is the target word and w _m is the noun co-occurring with the strongest association strength with w _t within a window size of ±9. In Cavallin (Reference Cavallin2012), the context word is a noun or a verb that forms verb-object relation with the target word. Mitra et al. (Reference Mitra, Mitra, Maity, Riedl, Biemann, Goyal and Mukherjee2015) use bigram relationship to specify the context word.

Within this group, various methods are proposed to obtain global semantics of the target word. In Tang et al. (Reference Tang, Qu and Chen2016), the global semantics of w _t is summarized as the Average Mutual Information (Entropy) of all possible senses in the corpus, as denoted below:

(1) $$\begin{eqnarray} {\zeta ^{{w_t}}} &=& \left[ {\begin{array}{@{}*{1}{c}@{}}C\\ {P\left( C \right)} \end{array}} \right] = \left[ {\begin{array}{@{}*{1}{c}@{}}{c_1},\ {c_2}, \ldots ,{c_i}, \ldots {c_n}\\ {p\left( {{c_1}} \right),\ p\left( {{c_2}} \right),\ \ldots ,p\left( {{c_i}} \right), \ldots p\left( {{c_n}} \right)} \end{array}} \right]\nonumber \\ & =& H\left( C \right) = - \sum \nolimits _{i = 1}^n {p\left( {{c_i}} \right)\log p\left( {{c_i}} \right)} \end{eqnarray}$$

Fig. 2.

Word co-occurrence network for the word space (Mitra et al. Reference Mitra, Mitra, Maity, Riedl, Biemann, Goyal and Mukherjee2015).

Positive Point Mutual Information (Bullinaria and Levy Reference Bullinaria and Levy2007; Bullinaria and Levy Reference Bullinaria and Levy2012) is also used in the literature to obtain global semantics. With this approach, the global semantics of the target word is obtained by computing the Point Mutual Information (PMI) I(w _c; w _t) = log(p(w _c|w _t)/p(w _c)) between the target word w _t and words in its context w _c. It is called Positive PMI because only those w _c with I(w _c; w _t) > 0 are kept. According to Bullinaria and Levy (Reference Bullinaria and Levy2012), keeping those with I(w _c; w _t) > 0 helps keep the words co-occurring more frequently with the target word and get rid of those with low frequency.

The major advantage of using explicit word vectors for global semantics is that these vectors are conveniently interpretable and directly comparable across different time intervals. Nevertheless, because the dimensions of these vectors are often very high, the problem of data sparseness cannot be avoided, which may lead to poorer performance compared with word-embedding approaches.

4.2 Approaches based on word-embedding

Approaches based on word-embedding map high-dimensional word vectors to low-dimensional vectors to obtain global semantics. According to the scope of co-occurrence considered in the operation, these approaches can be further divided into two subtypes: those relying on topic modelling and those relying on local word context.

The approaches relying on topic modelling such as Latent Semantic Indexing (LSA) (Deerwester et al. Reference Deerwester, Dumais, Furnas, Landauer and Harshman1990) and Latent Dirichlet Allocation (LDA) (Blei, Ng, and Jordan Reference Blei, Ng and Jordan2003) use co-occurrences of words within one document to characterize word semantics. Take LDA as an example. The main idea of LDA is that documents are represented as random mixture over latent topics where each topic is characterized by a distribution over words (Blei et al. Reference Blei, Ng and Jordan2003). This idea could be summarized by the following equation:

(2) $$\begin{equation} p\left( {w|d} \right) = p\left( {w|t} \right)*p(t|d) \end{equation}$$

Topic modelling can be extended to incorporate time to model the evolution of topics over time, represented by models such as Topic-Over-Time model (TOT) (Wang and Mccallum Reference Wang and Mccallum2006) and dynamic topic models (Blei and Lafferty Reference Blei and Lafferty2006). TOT is used in Wijaya and Yeniterzi (Reference Wijaya and Yeniterzi2011) for DWSC. It is a time-dependent extension of LDA that parameterizes a continuous distribution over time associated with each topic. By thus doing, the topics also take on information of time.

The SCAN model proposed by Frermann and Lapata (Reference Frermann and Lapata2016) is a dynamic Bayesian topic model, described in Figure 3. For each target word, a SCAN model is created that represents the word’s meaning with two distributions: a K-dimensional multinomial distribution over word senses ϕ and a V-dimensional distribution over the vocabulary ψ^k for each word sense. As a generative model, SCAN starts with a precision parameter κ^ϕ, a Gamma distribution with parameters a and b that controls the extent of meaning change of ϕ. A three time step model involves the generation of three sense distributions ϕ^{t − 1}, ϕ^t and ϕ^{t + 1}, at times <m >t-1, t < /m > and t + 1. The sense distribution at each time generates a sense z, which in turn generates the context words for the target word. Simultaneously, for each time of <m >t-1, t < /m > and t + 1, the word-sense distributions ψ^{k, t − 1}, ψ^{k, t} and ψ^{k, t + 1} are generated, each of which also generates the context words for the target word. The prior is an intrinsic Gaussian Markov Random Field which encourages smooth change of parameters at neighbouring times. The inference is conducted with blocked Gibbs sampling.

Fig. 3.

Three time step SCAN model (Frermann and Lapata Reference Frermann and Lapata2016).

Word embedding techniques that rely on local context of target words include Word2vec (Mikolov et al. Reference Mikolov, Sutskever, Chen, Corrado and Dean2013), generalized SVD, Glove (Pennington, Socher, and Manning 2014) and Dynamic Bernoulli Embeddings (Rudolph and Blei Reference Rudolph and Blei2018) etc. The scope of context is restrained within a window size. For instance, the objective of the Skip-Gram model is to maximize the average log probability of the following:

(3) $$\begin{equation} \frac{1}{T}\sum \nolimits _{t = 1}^T {\sum \nolimits _{ - c \le j \le c,j \ne 0} {\log p({w_{t + j}}|{w_t})} } \end{equation}$$

Word vectors obtained through embedding techniques such as generalized SVD, Word2vec and Glove cannot be directly used for DWSC. Different from explicit word vectors obtained by PPMI, these techniques map original stochastic distributions of words to lower dimensions, which will preclude comparison of the same word across time (Hamilton et al. Reference Hamilton, Leskovec and Dan2016). Therefore, word vectors of different time intervals obtained through these techniques have to be aligned to become comparable. Various approaches are proposed for the purpose. Based on the assumption that word vector spaces are equivalent under linear transformation and that meanings of most words do not shift over time, Kulkarni et al. (Reference Kulkarni, Alrfou, Perozzi and Skiena2015) adopt a linear transformation ${W_{t\prime \mapsto t}}( w ) \in {\mathbb {R}^{d \times d}}$ that maps a word vector ϕ_t to ϕ_t′ by solving the following optimization problem:

(4) $$\begin{equation} {W_{t\prime \mapsto t}}\left( w \right) = {\rm {arg}}\;{\rm {mi}}{{\rm {n}}_W}\sum \nolimits _{{w_i} \in k - NN\left( {{\phi _t},\left( w \right)} \right)} {\left| {\left| {{\phi _{t\prime }}\left( {{w_i}} \right)W - {\phi _t}\left( {{w_i}} \right)} \right|} \right|_2^2} \end{equation}$$

${W^{( t )}} \in {\mathbb {R}^{d \times | V |}}$

(5) $$\begin{equation} {R^{\left( t \right)}} = {\rm {arg}}\;{\rm {mi}}{{\rm {n}}_{_{{Q^T}Q = I}}}{\left| {\left| {Q{W^{\left( t \right)}} - {W^{\left( {t - 1} \right)}}} \right|} \right|_F} \end{equation}$$

${R^{( t )}} \in {\mathbb {R}^{d \times d}}$

Similar to TOT model, there are approaches that combine local-context word embedding with time. Yao et al. (Reference Yao, Sun, Ding, Rao and Xiong2017) proposed a diachronic word embedding model that integrates the word embedding learning process and the alignment operation by solving the following joint optimization problem:

(6) $$\begin{eqnarray} &&{\min _{U\left( 1 \right), \ldots ,U\left( T \right)}}\frac{1}{2}\sum \nolimits _{t = 1}^T \left| {\left| {Y\left( t \right) - U\left( t \right)U{{\left( t \right)}^T}} \right|} \right|_F^2 + \frac{\lambda }{2}\sum \nolimits _{t = 1}^T \left| {\left| {U\left( t \right)} \right|} \right|_F^2 \nonumber \\ &&\quad +\, \frac{\tau }{2} \sum \nolimits _{t = 2}^T {\left| {\left| {U\left( {t - 1} \right) - U\left( t \right)} \right|} \right|_F^2} \end{eqnarray}$$

Fig. 4.

Graphic representation of a dynamic embedding for text data in T time intervals, X ⁽¹⁾,. . ., X ^(T) (Rudolph and Blei Reference Rudolph and Blei2018).

There are also word-embedding-based studies that make use of both topic modelling and local word-context information, such as the semantic density model proposed in Sagi et al. (Reference Sagi, Kaufmann and Clark2009). This study first obtained word vectors by applying Latent Semantic Indexing (Deerwester et al. Reference Deerwester, Dumais, Furnas, Landauer and Harshman1990) on a term-term matrix of size 40,000 × 2,000 (40,000 words as vocabulary, 2,000 words as content words) filled with weighted co-occurrence counts of words, resulting in a reduced 40,000 × 100 matrix with each item in the vocabulary associated with a 100-dimension vector. Given a target word w _t, the instance sense of w _t is the context vector which is the normalized sum of the vectors associated with the words occurring with w _t in the instance sentence within the window size ±15. The global semantics of w _t is denoted by the density or the average pairwise similarity of the instance senses collected from the given corpus. The similarity of two instance senses is measured by cosine similarity.

The above review shows that many WSI techniques have already been used for DWSC. However, the state-of-the-art of WSI is not fully exploited in the literature of semantic change computation. There is currently no comprehensive comparison of these WSI approaches to DWSC, partly due to the fact that there is no large scale evaluation data available for the purpose. Further research is needed to understand the nature of these techniques. In addition, the literature also shows that not all WSI techniques are tested in semantic change computation, particularly those graph-based WSI techniques (Véronis Reference Véronis2004; Agirre and Soroa Reference Agirre and Soroa2007; Korkontzelos and Manandhar Reference Korkontzelos and Manandhar2010; Marco and Navigli Reference Marco and Navigli2013) that are reported to achieve state-of-the-art performance in Word Sense Disambiguation evaluation tasks. Moreover, while word-embedding models used in the current literature generally associate each word with a single embedding, the use of multisense embedding models (Kintsch Reference Kintsch2001; Reisinger and Mooney Reference Reisinger and Mooney2010; Li and Dan Reference Li and Dan2015; Liu et al. Reference Liu, Liu, Chua and Sun2015; Cao et al. Reference Cao, Huang, Ji, Chen and Li2017) is not yet explored in DWSC.

5.1 Metaphysical understandings of semantic change

The literature has reported two types of metaphysical understanding of change: Cambridge Change (Geach Reference Geach1969 : 71–72; 1979; Mortensen Reference Mortensen2016) and McTaggart Change (Broad Reference Broad1938 : 297). The definition of Cambridge Change is given as follows:

Many ordinary thoughts of change such as change of color from ‘red’ to ‘blue’ and change of weather from ‘hot’ to ‘cold’ are of this type. It implies an intrinsic difference at two points along the dimension of time. The metaphysical understanding of McTaggart Change originates from John McTaggart, which is defined below:

In McTaggart’s view, a change occurring at two points of time does not constitute a change. It establishes itself as a change when two successive events have different qualities. Both Cambridge Change and McTaggart Change agree that different qualities lead to change. They differ in how persistent the quality difference lasts. Cambridge Change is justified so long as the difference is detected at two distinctive points of time, while McTaggart Change maintains that change is determined should the difference exist in the successive course of time, but not just two slices of time.

Whether to adopt Cambridge Change or McTaggart Change in studies of semantic change relates to an important concept in language—convention. A conventionalized regularity is a regularity known and accepted at communal level and observed in almost any instance where the regularity applies (Lewis Reference Lewis1969 : 42). Because Cambridge Change considers only two points in the course of time, the semantic change based on such metaphysical understanding does not consider the course of time and therefore does not take convention into serious consideration. McTaggart Change, however, is based on events happening in the course of time and therefore incorporates conventionalization in its study.

Correspondingly in lexicography, convention serves to distinguish two types of semantic change: institutionalization and topicalicality (Fischer Reference Fischer1998 : 16). Institutionalization refers to the process by which a semantic change is integrated into language norm and becomes a convention, while topicality refers to the phenomenon that a word is used in connection with current affairs for a short period. For instance, Landau (Reference Landau2001) discussed the example chat group and speculated that the phrase may disappear in a couple of years with development of technology. Therefore, the phrase chat group is more of topicalicality than of institutionalization. From this perspective, Cambridge Change and McTaggart Change cover different semantic change phenomena. Cambridge Change covers both topicalicality and institutionalization, while McTaggart Change has a narrower scope and mainly concerns the institutionalization process that involves conventionalization.

Studies in theoretical semantic change generally adopt McTaggart Change as the metaphysical understanding of semantic change (Andersen Reference Andersen, Breivik and Jahr1989; Traugott and Dasher Reference Traugott and Dasher2002; Fortson Reference Fortson, Joseph and Janda2008). In the literature of semantic change computation, both Cambridge Change and McTaggart Change are adopted.

5.2 Computation models based on Cambridge change

Semantic change computation based on Cambridge Change mainly relies on similarity comparison. Given ζ_t1 and ζ_t2 as the global semantics of the target word w at time t ₁ and t ₂, the semantic change of w is indicated by the similarity measure as follows:

(7) $$\begin{equation} \theta = {\rm {Similar}}\left( {{\zeta _{t1}},\ {\zeta _{t2}}} \right)\ \end{equation}$$

(8) $$\begin{equation} \cos \left( {{\zeta _{t1}},\;{\zeta _{t2}}} \right) = \frac{{{\zeta _{t1}}\cdot {\zeta _{t2}}}}{{\left| {{\zeta _{t1}}} \right|\left| {{\zeta _{t2}}} \right|}} \end{equation}$$

(9) $$\begin{equation} {\rm {Nov}}\left( w \right) = \frac{{{\zeta _{t2}} - {\zeta _{t1}}}}{{{\zeta _{t1}}}} \end{equation}$$

Table 3.

Illustration of centroid-based distance (Abridged from Dubossarsky et al. (Reference Dubossarsky, Tsvetkov, Dyer, Weinshall and Grossman2015))

Cambridge Change can be used to detect novel senses. Researchers have proposed various ways to compute sense novelty on the basis of the ratio of sense distribution, as shown in Table 4.

Table 4.

Various ways to compute sense novelty

The major concern facing Cambridge-Change-based models is conventionality. Comparison of global semantics of two time intervals can answer the question whether a semantic change occurs or not, but it cannot tell whether the change has been conventionalized. Mitra et al. (Reference Mitra, Mitra, Maity, Riedl, Biemann, Goyal and Mukherjee2015) showed that the comparison method flagged 2,498 candidates with new semantic senses, but only 537 of them had been observed to be stable.

5.3 Computation models based on McTaggart Change

The models based on McTaggart Change rely on time series to investigate semantic change. Most studies in the literature fall into this group, as seen in Frermann and Lapata (Reference Frermann and Lapata2016), Yang and Kemp (Reference Yang and Kemp2015), Hilpert and Perek (Reference Hilpert and Perek2015), Jatowt and Duh (Reference Jatowt and Duh2014), Wijaya and Yeniterzi (Reference Wijaya and Yeniterzi2011), Tang et al. (Reference Tang, Qu and Chen2016) and many others. There are generally two ways to construct the time series. One way is to use the explicit global semantics of words obtained from corpora to form a time series, as follows:

(10) $$\begin{equation} \xi _{1,n}^w = \left[ {{\zeta _{{t_1}}},\ {\zeta _{{t_2}}},\ {\zeta _{{t_3}}},\ \ldots ,\ {\zeta _{{t_n}}}} \right] \end{equation}$$

(11) $$\begin{equation} \xi _{1,n}^w = [\Delta \left( {{\zeta _{{t_2}}},{\zeta _{{t_1}}}} \right),\;\Delta \left( {{\zeta _3},{\zeta _{{t_1}}}} \right), \ldots ,\;\Delta \left( {{\zeta _{{t_n}}},{\zeta _{{t_1}}}} \right)\rbrace \end{equation}$$

With McTaggart Change, it is possible to apply analyzing techniques of time series to semantic change computation. For instance, Sample Autocorrelation Function (sample ACF)Footnote ³ can be used to assess whether there is a trend of change in the time series and whether the trend consists of different stages. For illustration, given the time series data for the Chinese word tou ⁵¹ming ³⁵ from Tang et al. (Reference Tang, Qu and Chen2016) in Figure 5Footnote ⁴, the absolute values of sample ACF in Figure 6 show two sequences of decay from year lag 0 to 10 and from 13 to 20, indicating a change consisting of two stages.

Fig. 5.

Time series data for the Chinese word tou ⁵¹ming ³⁵.

Fig. 6.

Sample ACF with 0 ⩽ lag ⩽ 20 of the time series in Figure 5 with boundary $\pm 1.96/\sqrt{59}$. Data with Sample ACF below $\pm 1.96/\sqrt{n}$ (n is the number of data) is considered to contain no trend, since 1.96 is the 0.975 quantile of the standard normal distribution (Brockwell and Davis Reference Brockwell and Davis2002: 20).

Compared with Cambridge-Change-based models, the models based on McTaggart Change incorporate conventionality in semantic change computation, which enables more elaborated studies such as semantic change categorization.

As discussed in Section 4, different methods are used to obtain global semantics in DWSC. The global semantics obtained by these methods possesses different statistic nature and provides for different types of categorization. The global semantics obtained with LSA (as in Jatowt and Duh (Reference Jatowt and Duh2014)), PPMI (as in Hamilton et al. (Reference Hamilton, Leskovec and Dan2016) and Dubossarsky et al. (Reference Dubossarsky, Grossman and Weinshall2017)), or Skip-Gram model (as in Yao et al. (Reference Yao, Sun, Ding, Rao and Xiong2017) and Hamilton et al. (Reference Hamilton, Leskovec and Dan2016)) does not distinguish between word senses. Although differences obtained by contrasting global semantics of different time intervals can be used for semantic change detection, it does not tell whether the scope of word usage has increased or decreased. Therefore, global semantics of this nature cannot be used for more granulated categorization such as sense broadening and sense narrowing.

If the global semantics of a word contains information of different senses of target words, such as the entropy used in Tang et al. (Reference Tang, Qu and Chen2016) or word vectors obtained with LDA (Rohrdantz et al. 2011; Lau et al. Reference Lau, Cook, McCarthy, Newman and Baldwin2012), TOT (Wijaya and Yeniterzi Reference Wijaya and Yeniterzi2011) and SCAN (Frermann and Lapata Reference Frermann and Lapata2016), machine learning techniques can be used to identify semantic change categories such as sense broadening, sense narrowing, pejoration, amelioration, even metaphorical and metonymic change. For instance, Tang et al. (Reference Tang, Qu and Chen2016) discussed how to make use of curve fitting and Support Vector Machine to identify semantic change categories.

It is odd that models of semantic change are not much discussed in the literature, although many studies have adopted the McTaggart-Change-based approaches. One model, namely the S-Shaped Curve, is detailed in Tang et al. (Reference Tang, Qu and Chen2016). The S-Shaped Curve is defined by the logistic function below:

(12) $$\begin{equation} \zeta = \frac{{{e^{k + st}}}}{{1 + {e^{k + st}}}} \end{equation}$$

In addition, conventionality is also not much discussed in the literature, although the McTaggart-Change-based models enable such investigation. A new word usage gains its status as a novel sense only when it is conventionalized as well as being frequently used. Tang et al. (Reference Tang, Qu and Chen2016) discuss the issue very briefly by proposing to compute the conventionality of a word sense with the following:

(13) $$\begin{equation} {\rm {Conventionality}}_s^T = {\rm {AveragePrevalence}}\left( {s,T} \right) \times {\rm {DiaSpan}}\left( {s,T} \right) \times {\rm {DiaDensity}}\left( {s,T} \right) \end{equation}$$

In sum, the McTaggart-Change-based approaches to semantic change use time series to model semantic change. Although more information is available with the approaches, there is still a need to explore how semantic change should be modelled and how computational models relate to those hypotheses proposed in theoretical linguistics.

6.1 Categories of semantic change

In theoretical linguistics, there are different categorization schemes of semantic change (cf. Geeraerts (Reference Geeraerts1983), Blank and Koch (Reference Blank, Koch, Blank and Koch1999), Bloomfield (Reference Bloomfield1933) and many others). The categorization scheme proposed in Bloomfield (Reference Bloomfield1933) is one of the most popular schemes. We propose to formalize this scheme in terms of denotation change, as illustrated in Figure 7. In five of the six categories in the figure, the original sense of the word is retained. The original sense can be widened (thus, the category of widening) or narrowed (thus, the category of narrowing). New senses can be added to the word. The added sense can be of the same semantic domain (thus, the category of metonymy) or of different domains (thus, the category of metaphor). The category of semantic shift specifies the cases in which a new sense is substituted for its original one. Along with the sense substitution, the emotional colouring of the word might be changed from commendatory to derogatory (thus, the category of pejoration) or from derogatory to commendatory (thus, the category of amelioration).

Fig. 7.

Schematic diagram of semantic change categories for theoretical exploration.

The phenomena of widening and narrowing are discussed in most studies, such as Sagi et al. (Reference Sagi, Kaufmann and Clark2009) and Gulordava and Baroni (Reference Gulordava and Baroni2011). Novel senses, namely the birth of lexical senses are discussed in details in Erk (Reference Erk2006), Lau et al. (Reference Lau, Cook, McCarthy, Newman and Baldwin2012), Mitra et al. (Reference Mitra, Mitra, Maity, Riedl, Biemann, Goyal and Mukherjee2015), Frermann and Lapata (Reference Frermann and Lapata2016), Tang et al. (Reference Tang, Qu and Chen2016) and others. Pejoration and amelioration are dealt with in Jatowt and Duh (Reference Jatowt and Duh2014). Using words from SentiWordNetFootnote ⁵ as constitutes in context, Jatowt and Duh (Reference Jatowt and Duh2014) conduct case studies of aggressive, propaganda and fatal in Google Books Ngram Corpus and show that aggressive has undergone amelioration and that fatal and propaganda have undergone pejoration. Semantic shift is discussed in Wijaya and Yeniterzi (Reference Wijaya and Yeniterzi2011), with examples such as gay and awful. Detection of semantic change motivated by metonymy and metaphor is discussed in Tang et al. (Reference Tang, Qu and Chen2016).

The studies of semantic change can also add the ability of time awareness to NLP systems and help improve performance in disambiguating word senses. Along this track, one possible categorization scheme is proposed in Mitra et al. (Reference Mitra, Mitra, Maity, Riedl, Biemann, Goyal and Mukherjee2015). The research proposed four types of semantic change based on change in semantic clusters: split, join, birth and death. Because splitting one cluster into two and joining of two clusters are more of the results of clustering algorithm than of semantic change, we propose to modify the category schema into three categories: nonchange, birth and death, as illustrated in Figure 8.

Fig. 8.

Schematic diagram of semantic change categories for NLP (adapted from Mitra et al. (Reference Mitra, Mitra, Maity, Riedl, Biemann, Goyal and Mukherjee2015)).