1 Introduction
Named entity recognition (NER) is an important task in natural language processing that aims to discover references to entities in text. The task was first introduced in the sixth Message Understanding Conference (MUC-6) (Grishman and Sundheim Reference Grishman and Sundheim1996) as a short-term subtask. At that time, it was thought that a practical system could be developed in a relatively short time, which could also serve as a domain-independent tool for other information extraction tasks. To accomplish the NER task, portions of text were selected so that the selected text refers to the same entity in all possible contexts in which they exist, and does not refer to anything else in contexts in which that entity does not exist, thus being rigid designators (Kripke Reference Kripke1982).
The following examples exemplify the task and give hints about the complexity of the task. First of all, a single entity might be referred with various phrases: the terms ‘JFK’, ‘Kennedy’ or ‘John F. Kennedy’ may all refer to the same entity in relevant contexts – the thirty-fifth president of the United States. Second, the phrases ‘JFK’ or ‘John F. Kennedy’ might refer to the airport in New York City. In the definition of the task made in MUC-6, the most prominent designators were person names, geographical locations and organization names, which became the simplest definition of the NER task.
The demand for NER in application domains such as of news outlets, social media and e-commerce hubs has further motivated research in this area (see Section 2). In a recent work, long short-term memory (LSTM) and gated recurrent units have been utilized for this task (Huang, Xu and Yu Reference Huang, Xu and Yu2015; Lample et al. Reference Lample, Ballesteros, Subramanian, Kawakami and Dyer2016; Ma and Hovy Reference Ma and Hovy2016; Yang, Salakhutdinov and Cohen Reference Bhatia, Guthrie and Eisenstein2016). However, these approaches are not well studied for morphologically rich languages (MRLs). MRLs retain information in the morphology of the surface form of words, which is present in the syntax and word n-grams in other languages. For example, in Turkish, a MRL, the word ‘İstanbul’daydı’ means ‘he/she was in Istanbul’, which embodies the tense and the locative case. This makes morphological understanding more important in comparison to languages with simpler morphological mechanisms. Thus, a considerable amount of research has been undertaken to understand the morphological properties in these languages.
The motivation behind this study can be explained using the same Turkish example. The morphological analysis of the word ‘İstanbul’daydı’ is ‘İstanbul+Noun+Prop+A3sg+Pnon+Loc^DB+Verb+Zero+Past+A3sg’. The notation shown in this representation of morphological information originates from the widely accepted representation scheme for Turkish (Oflazer Reference Oflazer1994). The analysis denotes that the root word ‘İstanbul’ is a proper noun in locative case and it is not marked with a possessive marker (see Section 3.1.1). The final word is a verb in past tense of third person singular agreement. One can suggest that the locative case tag and the fact that it is not possessed by anyone or anything might be a good indicator of being a named entity.
The hypothesis of this work is that morphological tags capture syntactic and semantic information and, thus, help in improving the NER performance. Thus, sequence tagging models that rely on LSTM networks are employed to provide evidence to this hypothesis and to solve the NER task for MRLs. One can argue that the literature already addressed this issue by proposing character-based embeddings in word representations (Lample et al. Reference Lample, Ballesteros, Subramanian, Kawakami and Dyer2016) and entities tagged at the character level (Kuru, Can and Yuret Reference Kuru, Can and Yuret2016). Moreover, it is possible to note that morphological tags have been employed in the past for the NER task (Tür, Hakkani-Tür and Oflazer Reference Oflazer2003; Yeniterzi Reference Yeniterzi2011). However, our work treats the morphological analysis in a number of different ways that can be applied to many MRLs and is the first to propose an embedding-based framework for representing the morphological analysis in the context of NER.
The main contributions of this work are a state-of-the-art system for NER in MRLs and providing evidence that reveals augmenting word representations with morphological embeddings improves NER performance.
The paper is organized as follows: Section 2 summarizes related work. Section 3 provides information about MRLs and Bi-LSTM networks utilized in the proposed model. Section 4 presents the proposed model. Section 5 presents the experiments in Turkish, Czech, Hungarian, Finnish and Spanish to evaluate our approach. In Section 5, the results of these experiments and a comparison of the proposed model’s state-of-the-art results with the previous work are also given. Section 6 makes concluding remarks.
2 Related work
NER is closely related to complex natural language understanding tasks such as relation extraction (Miwa and Bansal Reference Miwa and Bansal2016), knowledge base population (Rao, McNamee and Dredze Reference Rao, McNamee, Dredze, Poibeau, Saggion, Piskorski and Yangarber2013) and question answering (Lee, Hwang and Jang Reference Lee, Hwang and Jang2007; Liu and Ren Reference Liu and Ren2011; Lee et al. Reference Lee, Kim, Yoo, Jung, Kim and Yoon2017). Furthermore, NER systems are often part of search engines (Guo et al. Reference Guo, Xu, Cheng and Li2009) and machine translation systems (Babych and Hartley Reference Babych and Hartley2003).
Early studies proposed compiling lists of people, place and organization names and exploiting them to decide whether there exists a named entity using hand-crafted rules (Appelt et al. Reference Appelt, Hobbs, Bear, Israel, Kameyama, Martin, Myers and Tyson1995; Humphreys et al. Reference Humphreys, Gaizauskas, Azzam, Huyck, Mitchell, Cunningham and Wilks1998). Traditional approaches typically use several hand-crafted features such as capitalization, word length, gazetteer related features and syntactic features (part-of-speech (POS) tags, chunk tags, etc.). A wide range of machine learning-based methods have also been proposed to address the NER task. Some of the well-known approaches are conditional random fields (CRFs) (McCallum and Li Reference McCallum and Li2003; Finkel, Grenager and Manning, Reference Finkel, Grenager and Manning2005), maximum entropy (Borthwick Reference Borthwick1999), bootstrapping (Jiang and Zhai Reference Jiang and Zhai2007; Wu et al. Reference Wu, Lee, Ye and Chieu2009), latent semantic association (Guo et al. Reference Guo, Xu, Cheng and Li2009) and decision trees (Szarvas, Farkas and Kocsor Reference Szarvas, Farkas, Felföldi, Kocsor and Csirik2006). These techniques are generally used to create classification models that act on every token to decide whether there is an entity on that position of the text or not.
Recently, deep learning models have been instrumental in deciding how the parts of the input should be composed to form the most beneficial features leading to state-of-the-art results (Collobert et al. Reference Collobert, Weston, Bottou, Karlen, Kavukcuoglu and Kuksa2011). One of the key issues is the determination of how to represent the words. This is due to the symbolic nature of words. These methods rely on simple tokenization by white space and employ distributional hypothesis (Harris Reference Harris1954). The research in this direction led to the use of fixed-length vectors in a dense space that improved the overall performance of many tasks, such as sentiment analysis (Socher et al. Reference Socher, Perelygin, Wu, Chuang, Manning, Ng and Potts2013), syntactic parsing (Collobert and Weston Reference Collobert and Weston2008), language modeling (Mikolov et al. Reference Mikolov, Karafiát, Burget, Cernocky and Khudanpur2010), POS tagging and NER (Collobert et al. Reference Collobert, Weston, Bottou, Karlen, Kavukcuoglu and Kuksa2011). These word representations or embeddings are automatically learned either during or before the training phase using methods such as Word2Vec (Mikolov et al. Reference Mikolov, Sutskever, Chen, Corrado and Dean2013), GloVe (Pennington, Socher and Manning Reference Pennington, Socher and Manning2014) and fastText (Bojanowski et al. Reference Bojanowski, Grave, Joulin and Mikolov2017). The incorporation of morphology into this type of word embeddings was proposed for language modeling (Luong, Socher and Manning Reference Luong, Socher and Manning2013; Santos and Zadrozny Reference Santos and Zadrozny2014; Bhatia, Guthrie and Eisenstein Reference Bhatia, Guthrie and Eisenstein2016; Lankinen et al. Reference Lankinen, Heikinheimo, Takala, Raiko and Karhunen2016; Xu and Liu Reference Xu and Liu2017) and for morphological tagging and segmentation (Shen et al. Reference Shen, Clothiaux, Tagtow, Littell and Dyer2016; Cotterell and Schütze 2017).
Built upon these findings, new approaches have been proposed that treat the NER task as a sequence labeling problem (Huang, Xu and Yu Reference Huang, Xu and Yu2015; Lample et al. Reference Lample, Ballesteros, Subramanian, Kawakami and Dyer2016; Ma and Hovy Reference Ma and Hovy2016; Yang et al. Reference Yang, Salakhutdinov and Cohen2016). These studies employ LSTM or gated recurrent unit components to capture the syntactic and semantic relations between the units that make up a natural language sentence. In Huang et al. (Reference Huang, Xu and Yu2015), a Bi-LSTM network with a CRF layer on label scores is proposed. A special fixed-size representation is prepared for each word. The first component of this representation is the spelling features extracted from the surface forms such as whether all letters are lowercase or whether the first letter is uppercase. The second component is composed of a feature for each word bi-gram and tri-gram that exists in the context. The resulting representation is then fed to a word-level LSTM. This network’s NER performance is reported to be comparable to the performance of the state-of-the-art studies. Another similar approach processes characters in each surface form in the sentence with a convolutional neural network (Ma and Hovy Reference Ma and Hovy2016). This helps the network to automatically represent the features that are extracted according to rules that are carefully designed in the previous work. The word representations formed with this additional component are then fed to a word-level LSTM resulting in state-of-the-art performance. Two different approaches use LSTMs (Lample et al. Reference Lample, Ballesteros, Subramanian, Kawakami and Dyer2016) and gated recurrent units (Yang et al. Reference Yang, Salakhutdinov and Cohen2016) instead of convolutional neural networks at the character and word levels otherwise similar to other work (Ma and Hovy Reference Ma and Hovy2016). These two studies report results similar to each other and improve the state-of-the-art results.
There are previous studies that give special importance to MRLs for the NER task. Tür et al. (Reference Tür, Hakkani-Tür and Oflazer2003) employed the last inflectional group in a statistical model that relies on maximum a posteriori estimation for Turkish. In another work, morphemes in a morphological analysis are used as features of a CRF model (Yeniterzi Reference Yeniterzi2011). Şeker and Eryiğit (Reference Demir and Özgür2012) also employed a CRF model with more extensive features and added gazetteers. Demir and Özgür (Reference Demir and Özgür2014) used similar features for a CRF model with the addition of word embeddings. In our work, however, the disambiguated morphological analysis is used to generate a fixed-length vector in a number of different ways (Section 4.2), which is called a morphological embedding. This morphological embedding is composed with pretrained word embeddings and character-based embeddings to obtain a word representation for each word and employed in a setting similar to the previous work (Ma and Hovy Reference Ma and Hovy2016).
3 Background
This section provides background information about the MRLs examined in this work and the bidirectional LSTM (Bi-LSTM) model that is used for sequence tagging for NER.
3.1 Morphologically rich languages
The languages in which some of the syntactical functions of words are expressed by morphological phenomena within the surface forms of the words are called MRLs. In these languages, the number of words that can be generated from a single root word is very high in general. A significant number of inflections and derivations are possible for most root nouns and verbs. In practice, however, this potential is not fully realized where a few of the affixes are attached to a stem in succession to form new words. Regardless, the number of words that can be obtained from the root words is very large. This expressiveness gives rise to complications in applications, such as data sparseness due to words with many alternative affixes.
The following sections describe the basic characteristics of the MRLs Turkish, Czech, Hungarian, Finnish and Spanish.
3.1.1 Turkish
Turkish is an agglutinative language, which expresses most syntactic information through the morphology of the surface form of the words. Thus, studies in Turkish natural language processing have mainly focused on the morphological analysis of words. To this end, a finite state transducer based on a two-level formalism (Koskenniemi Reference Koskenniemi1983) was introduced (Oflazer Reference Oflazer1994) to capture the rules of Turkish morphology (Underhill Reference Underhill1976; Lewis Reference Lewis1991). The notation introduced in this work is considered the standard for morphological analysis in Turkish. The morphological analysis of the word ‘İstanbul’daydı’ (he/she was in İstanbul) is as follows:
where ‘Prop’ indicates a proper noun, ‘A3sg’ denotes the third singular person agreement and ‘Pnon’ signifies that no possessive agreement is active. ‘DB’ (derivational boundary) indicates a transition of the POS usually induced by a derivational suffix. It also marks the beginning of a new sequence of inflectional morphemes called inflectional group (Oflazer Reference Oflazer2003). In this example, the derivation is triggered by the ‘-ydı’ suffix, which is decoded with the ‘Past’ (past tense) tag.
3.1.2 Finnish
Finnish is an agglutinative language that exhibits vowel harmony and consonant gradation. Finnish uses derivational suffixes to a great extent. In our work, the morphological tagging of Finnish text is done by a tool called FinnPos (Silfverberg et al. Reference Silfverberg, Ruokolainen, Lindén and Kurimo2016), which is an averaged structured perceptron classifier. FinnPos relies on Omorfi (Pirinen Reference Pirinen2014) for morphological labels and lemmatization. In this work, the tool named ftb-label from the FinnPos package was used to obtain the disambiguated analysis as the morphological information associated with the word.
An example output of the system for the word ‘Tampereella’ (in Tampere) is as follows:
which tags the word as a singular proper noun in adessive case.
3.1.3 Czech
Czech is a language with free word order known for its rich morphological properties. The morphological tags for Czech in our work is given in the NER dataset by Ševčíková et al. (2007), which is actually a subset of the Prague Dependency Treebank 2.0 (Hajič et al. Reference Hajič, Panevová, Hajičová, Sgall, Pajas, Štěpánek, Havelka, Mikulová, Žabokrtský, Ševčíková-Razímová and Urešová2006). These tags consist of fifteen character string where each position encodes a different morphological aspect. The tags in this treebank were labeled by seven annotators in two phases. In the first phase, annotators were given the output of a morphological tagger and were requested to select the best option. In the second phase, the discrepancies in annotator responses were resolved by another person (Hajič et al. Reference Hajič, Hajičová, Mikulová, Mírovský, Ide and Pustejovsky2017). For example, the word ‘dlaně’ (palms) is decoded as follows:
which tags the word as a common noun, feminine, plural, genitive and not-negated. The tag can be decoded by a simple lookup in the tables provided in the morphological annotation manual for the Prague Dependency Treebank 2.0 (Hana et al. Reference Hana, Zeman, Hajic, Hanová, Hladká and Jerábek2005).
3.1.4 Hungarian
Hungarian is also a MRL with free word order. In this work, the morphological features produced by the magyarlanc tool are associated with words (Zsibrita, Vincze and Farkas Reference Zsibrita, Vincze and Farkas2013). There are three main morphological coding schemes for Hungarian: Humor (Proszeky and Tihanyi Reference Proszeky and Tihanyi1993), MSD (Erjavec Reference Erjavec2010) and KR (Tron et al. Reference Tron, Halacsy, Rebrus, Rung, Simon and Vajda2006). magyarlanc uses a harmonization of these schemes (Farkas et al. Reference Farkas, Szeredi, Varga and Vincze2010), which is also adopted in this work.
As an example, the morphological features for ‘nyelvészek’ (linguists) is as follows:
which indicates the nominal case and plurality.
3.1.5 Spanish
Spanish is a Romance language that is not generally considered as a MRL. We examine it along with MRLs because it differs from many languages as the number of conjugated forms per verb can be as high as forty-seven. A simplified version of the POS tags from the AnCora treebankFootnote 1 was used to obtain the morphological tags of the words. Here, tags do not specify the morphological features separately (as in the other four languages); they carry morphological information attached to the word. For instance, the Spanish word ‘visitada’ is labeled as ‘VMP’, indicating the past participle form of the verb. Here, ‘V’ indicates a verb, ‘M’ states that the verb is principal and ‘P’ indicates a participle verb.
For sentences within the corpus, the supplied POS tags are used, while for sentences outside the corpus, the Stanford POS tagger is employed.
3.2 Bidirectional LSTMs
Bi-LSTM is a type of sequential neural network that is composed of two LSTM modules, in which the first one reads the input sequence in the forward order and the second one reads it in the reverse order. The backwards component is important as it captures information about subsequent words, which can be highly significant in natural language processing tasks.
LSTM models were introduced to solve the vanishing gradients problem of recurrent neural networks (Hochreiter and Schmidhuber Reference Hochreiter and Schmidhuber1997). The sequential formulation of recurrent neural network is defined in terms of the following functions given a sequence of input vectors x t of size d:
\begin{align*}
h_t & = \text{tanh} (U x_t + W h_{t-1}) \\
o_t & = \operatornamewithlimits{softmax}(V h_{t}) \\
\end{align*}
where h t is the hidden state corresponding to item t of size p, o t is the output vector corresponding to item t of size p, U is a p × d matrix and W, V are matrices of size p × p.
LSTM differs from recurrent neural network in how it computes the output and hidden vectors. The following equations define an LSTM architecture (Hochreiter and Schmidhuber Reference Hochreiter and Schmidhuber1997):
\begin{align*}
i_t & = \sigma (W^{(i)} x_t + U^{(i)} h_{t-1}) \\
f_t & = \sigma (W^{(f)} x_t + U^{(f)} h_{t-1}) \\
o_t & = \sigma (W^{(o)} x_t + U^{(o)} h_{t-1}) \\
\tilde{c}_t & = \text{tanh} (W^{(c)} x_t + U^{(c)} h_{t-1}) \\
c_t & = f_t \circ c_{t-1} + i_t \circ \tilde{c}_t \\
h_t & = o_t \circ \text{tanh} (c_{t}) \\
\end{align*}
where σ is the sigmoid function and ○ is the element-wise multiplication. In this architecture, variables i t, f t and o t represent the will of the LSTM to memorize the cell’s new value 
$\tilde{c}_t$, forget the contents of the previous cell’s value c t − 1 and display the current cell’s value c t, respectively. The information is carried by c t − 1 and h t − 1 from the previous item. In this architecture, the hidden state c t is calculated as a parameterized sum of the previous hidden state c t − 1 and $\tilde{c}_t$, which is a nonlinear function of the input x t and the previous output h t − 1. This eliminates the repetitive multiplication performed in the recurrent neural network architecture, thereby solving the vanishing gradient problem.
For a single LSTM cell, the output vector h t can be used for classification, regression or as input to upper layers of the neural network. In the case of Bi-LSTM, two LSTM cells, one for forwards and one for backwards, with their corresponding output vectors 
$\overrightarrow{h_t}$ and 
$\overleftarrow{h_t}$ are defined. The concatenation 
$h_t = [\overrightarrow{h_t}; \overleftarrow{h_t}]$ is the output of a Bi-LSTM component (Graves and Schmidhuber Reference Graves and Schmidhuber2005).
An LSTM cell is trained by learning the variables, the p × d matrices W (i), W (f), W (o), W (c) and the p × p matrices U (i), U (f), U (o), U (c). Learning is performed by backpropagation through time with an update rule of choice (i.e., vanilla stochastic gradient descent or a more advanced gradient-based algorithm). The training of Bi-LSTM is the same with additional parameters for the extra LSTM.
4 Model
In this work, the NER problem is treated as a sequence tagging problem where the model expects the input sentence to be labeled with its corresponding entities. Figure 1 shows a sentence that is labeled with the Inside Outside Beginning (IOB) scheme. The IOB tagging scheme labels the first token of an entity by prefixing its tag with ‘B-’, consecutive tokens of the same entity with ‘I-’ and tokens not associated with any entity simply as ‘O’, which is omitted in the figures for purposes of simplicity. So, the sample in the figure contains three named entities: ‘UN’ as an organization name, ‘Rolf Ekéus’ as a person name and ‘Baghdad’ as a location name.
NER sample with IOB tagging scheme.
The input sentence of length n is defined as X = (x 1, x 2, . . ., x n) and Y = (y 1, y 2. . ., y n). Each x i is a fixed-length vector of size d, consisting of embeddings that represent the ith word. y i is a vector of size K such that y ik = 1 if and only if the correct tag is the kth tag in our tag vocabulary of size K, otherwise y ik = 0. Further details of word representations are provided in Section 4.1. The words x i are then fed to a Bi-LSTM, which is composed of two LSTMs (Hochreiter and Schmidhuber Reference Hochreiter and Schmidhuber1997) treating the input forwards and backwards, respectively (see Section 3.2).
The forward and backward components’ cell matrices 
$\overrightarrow{H}$ and 
$\overleftarrow{H}$ are both of size n × p, where p is the number of dimensions of one component of the Bi-LSTM. Figure 2 describes how the proposed model works with a Turkish sentence as an example. 
$\overrightarrow{H}_{i,j}$ denotes the value of the jth dimension of the ith output vector of the forward component that corresponds to the ith word in the sentence, whereas the corresponding value in the backward component is denoted by 
$\overleftarrow{H}_{n-i+1,j}$. The concatenation of rows 
$\overrightarrow{H}_{i}$ and 
$\overleftarrow{H}_{n-i+1}$ from $\overrightarrow{H}$ and $\overleftarrow{H}$, respectively, are fed to a fully connected hidden layer of K output neurons for each of the n input words. The output of this fully connected layer for the ith word is denoted by ξi.
Processing a Turkish sentence with the proposed model. Words are represented with a fixed-length vector that combines the word, character and morphological embeddings (see Section 4.2).
A CRF (Lafferty, McCallum and Pereira Reference Lafferty, McCallum and Pereira2001) based approach is followed to predict the entity tags. CRF-based approaches model the dependencies between consecutive input units better than the approaches that only try to predict the correct tag based on ξiFootnote 2. The advantage of the CRF model stems from the logical requirements of the IOB tagging scheme. For instance, the scheme allows tags that start with ‘I-’ only after a ‘B-’ or ‘I-’ tag of the same type. That is, ‘I-PERSON’ might only come after ‘B-PERSON’ or ‘I-PERSON’, and not after ‘O’ or ‘B-LOCATION’ or others. Moreover, the sequences in the corpus might indicate an ordering relation between two tag types. For instance, ‘LOCATION’ tags may tend to appear more frequently before ‘PERSON’ tags compared to the other way around. Since CRF models allocate a probability to every valid sequence, it is possible to determine which tag sequences do not adhere to these rules as well as assigning higher probability to sequences that are in line with the ordering relations.
In order to implement a CRF model, the tag score vector ξi at each position i is treated as the observation score obtained from the fully connected layer and the following objective function for a sample sentence X is minimized:
$$\begin{equation}
s(X, y) = \sum _{i} A_{y_i,y_{i+1}} + \sum _i \xi _{i,y_i}
\end{equation}$$
$$
y^* = \operatornamewithlimits{argmax}_{y^{\prime }} s(X, y^{\prime }).
$$
4.1 Embeddings
It has been shown that modeling units of information in a natural language input as fixed-length vectors, called embeddings, is more effective at encoding semantic properties of the words compared to using manually designed features (Turian, Ratinov and Bengio Reference Turian, Ratinov and Bengio2010; Collobert et al. Reference Collobert, Weston, Bottou, Karlen, Kavukcuoglu and Kuksa2011). This work utilizes embeddings where the input words, x i, are represented as fixed-length vectors composed of three components: word, character and morphological.
Word embeddings: For each unique word, a vector of length d w is defined. As mentioned above, word representations are connected to the final loss expression. This indirect relation makes them parameters of the model. Thus, it is possible to optimize each of the d w dimensions for the target task for each distinct word. However, these parameters are not learned from scratch during training. The word embeddings are initialized to vectors obtained through approaches like skipgram with negative sampling (Mikolov et al. Reference Mikolov, Sutskever, Chen, Corrado and Dean2013) and fastText (Bojanowski et al. Reference Bojanowski, Grave, Joulin and Mikolov2017). If a corpus larger than the Wikipedia for a language is available, the skipgram algorithm is employed to obtain the word embeddings using that corpus. If such a corpus is not available, we use the word embeddings that are pretrained by Bojanowski et al. (2016) using Wikipedia.
Character embeddings: In addition to the word embedding for a word, the covert relationships in the character sequence of the word is of value (Ma and Hovy Reference Ma and Hovy2016). To capture these relationships, a separate Bi-LSTM component is used for this embedding type with a cell dimension of d c. This Bi-LSTM component is fed with the characters of the surface form of the ith word. After all the characters are processed by the Bi-LSTM component, the last cell’s output of the forward and backward LSTMs are concatenated to obtain the character embedding of the word of length 2d c (see Figure 3).
The representation of the Turkish word ‘evlerinde’ according to the proposed model featuring all three components: word, character and morphological embeddings.
Morphological embeddings: These embeddings are constructed similar to character embeddings. In this case, the tags of the morphological analysis are treated as a sequence and fed to a separate Bi-LSTM component for morphological embeddings, resulting in a vector of length 2d m. Several alternative combinations to serve as morphological tag sequences are described in Section 4.2.
When all of the components are active, the total size of a word representation is d = d w + 2d m + 2d c. Figure 3 shows the representation for the Turkish word ‘evlerinde’ (in their houses). This representation is then fed into the sentence-level Bi-LSTM described in Section 4.Footnote 3
4.2 Morphological embedding configurations
In order to determine an effective configuration for extracting the syntactic and semantic information in the morphological analysis of a word, experiments with a total of four different combinations of morphological tags were performed. In MRLs, it is common for a word to have more than one possible morphological analysis. The correct analysis within a context is determined using a morphological disambiguator for that language. For example, the Turkish word ‘evlerinde’ has three different meanings depending on the context: ‘in their house’, ‘in their houses’ and ‘in his/her houses’. If the correct sense in a particular context is the last one, then the disambiguator will output the morphological analysis ‘ev+Noun+A3pl+P3sg+Loc’. Here, ‘A3pl’ indicates third person plural, ‘P3sg’ is the possessive marker for third person singular and ‘Loc’ is the locative case marker. After the correct morphological analysis for a word is determined, the embeddings are formed as explained below.
Table 1 shows an example for each morphological embedding configuration employed in this work for each language. The first one uses the root accompanied with all the morphological tags in the analysis. This embedding configuration is called WITH ROOT (WR). This is the simplest embedding style that can be considered given the morphological analysis of words in any language. This is because most of the time the morphological tags output by morphological disambiguation tools can be treated as sequences.
Embedding configurations for Turkish, Czech, Hungarian, Finnish and Spanish: WITH ROOT (WR), WITHOUT ROOT (WOR), WITH ROOT AND AFTER LAST DB (WR_ADB), CHAR
For Turkish, the morphological analysis of the word ‘evlerinde’, which is ‘ev+Noun+A3pl+P3sg+Loc’, is transformed into a list by splitting from the ‘+’ symbols. For Czech, the output of the morphological disambiguator (Votrubec Reference Votrubec2006) are the root lemma and a string of fifteen characters for tags. In the WR configuration, this is transformed into a fixed-length list by splitting with the ‘+’ symbol. For instance, the analysis ‘prezident+NNMS1—–A—-’ is converted into ‘prezident’, ‘NNMS1—–A—-’. A similar transformation is applied to Hungarian. The analysis of the word ‘Magyar’, ‘Magyar+PROPN+Case=Nom+Number=Sing’, is converted into the list ‘Magyar’, ‘PROPN’, ‘Case=Nom’, ‘Number=Sing’. The disambiguator that is used for Finnish outputs the tags separately so it is just transformed into a list. The disambiguated morphological analysis of the word ‘Tampereella’ is ‘tampere+[POS=NOUN]+[PROPER=PROPER]+[NUM=SG]+[CASE=ADE]’. Then, for the embedding configurationWR, it is transformed into ‘tampere’, ‘[POS=NOUN]’, ‘[PROPER=PROPER]’, ‘[NUM=SG]’, ‘[CASE=ADE]’. One exception to this is Spanish in our choice of languages. For Spanish, a single tag is used to represent both the POS and the morphological properties that might be attached to the word. For instance, the Spanish word ‘visitada’ is only labeled with ‘VMP’, indicating the past participle form. So, the result is just a single item list for Spanish.
The second embedding configuration follows from the WITH ROOT (WR) scheme. The root is omitted from the list and the resulting list is the embedding configuration WITHOUT ROOT (WOR). Obviously, this embedding configuration does not make sense for languages that do not carry the root lemma in their morphological analysis representations, such as Spanish in our case.
The next configuration is specific to the Turkish language. The morphological notation of Turkish includes a ‘DB’ (derivational boundary) tag. This tag denotes a change in the POS during the derivational process. The parts that are separated by DB tags are called inflectional groups (Oflazer Reference Oflazer2003). To examine the significance of these boundaries on the performance of the task, the tags between the root and the last derivational boundary are removed. The intuition is that the information given by the features before the last derivational boundary may not be relevant at all or may be relatively less relevant. This is because the last derivational part yields the derived word whose lexical and syntactic properties may be slightly different from the intermediate parts. This configuration is named as WITH ROOT AND AFTER LAST DB (WR_ADB). Table 1 shows an example for the word ‘İstanbul’daydı’ (he/she was in İstanbul), whose analysis was given in Section 3.1.1.
Finally, the morphological analysis of a word is simply treated as a string, which is transformed to a list containing each of its characters. This embedding configuration is referred to as CHAR.
5 Experiments
To test the validity and performance of our proposed method, two main set of experiments are conducted: (i) experiments that compare the proposed approach with the state-of-the-art models, and (ii) experiments that aim to demonstrate the differing performance characteristics of different model configurations. In this section, before giving the results of the experiments, the training method and the datasets used in the experiments are explained.
5.1 Training
The parameters to be learned by the training algorithm are the parameters of the Bi-LSTM described in Section 4 (Figure 2), the parameters of the Bi-LSTMs for the character and morphological embeddings (Figure 3) and the word embeddings for each unique word. After experiments with several different choices for the number of dimensions for these parameters, a choice of 100 for word embeddings, 200 for character embeddings and 200 for morphological embeddings was observed to give the best results. However, it was not possible to use these dimension sizes in the experiments with all the languages and configurations due to time complexity.
In the first set of experiments that aim to compare the model configurations, the cell dimension of the sentence-level LSTM, word, character and morphological embedding dimensions and character and morphological LSTM cell dimensions were set as ten. For word embeddings, embeddings of size ten were trained with an algorithm that takes sub-word information into account (Bojanowski et al. Reference Bojanowski, Grave, Joulin and Mikolov2017) using the corresponding language version of Wikipedia and were used as pretrained word embeddings. Higher dimension sizes that yield the best performances as stated above were used in the second set of experiments that compare the proposed model with the previous work.
Model training was done by calculating the gradients using the back propagation algorithm and updating with the stochastic gradient descent algorithm with a learning rate of 0.01. Gradient clipping was employed to handle gradients diverging from zero. Additionally, dropout was used on the inputs with probability 0.5. Each language was trained for 50 epochs.
5.2 Datasets
Five languages were selected to evaluate the proposed method on a set of MRLs: Turkish, Czech, Hungarian, Finnish and Spanish. In this section, the specifics of each dataset like the morphological analysis format, the pretrained word embeddings we used and the origin of the data is given.
Turkish: Our model is trained and evaluated using a corpus which was widely used in previous works on Turkish NER (Tür et al. Reference Tür, Hakkani-Tür and Oflazer2003). The training part of the corpus contains 14,481 person names, 9,411 location names and 9,037 organization names while the test part contains 1,594 person names, 1,091 location names and 863 organization names. In addition to the named entity tags and the corresponding surface forms, the corpus also contains a single disambiguated morphological analysis for each input word.
Word embeddingsFootnote 4 of Turkish words as vectors of length 100 were obtained using the skipgram algorithm (Mikolov et al. Reference Mikolov, Sutskever, Chen, Corrado and Dean2013) on a corpus of 951 million words, 2,045,040 of which are unique (Yildiz et al. Reference Yildiz, Tirkaz, Sahin, Eren and Sonmez2016). This corpus consists of Turkish texts extracted from several national newspapers, news sites and book transcripts. The fastText algorithm was employed to obtain word embeddings of size ten using the same corpus (Bojanowski et al. Reference Bojanowski, Grave, Joulin and Mikolov2017).
Czech: CNEC 2.0 corpus was used to test the performance of our model on the Czech language (Ševčíková et al. Reference Ševčíková, Žabokrtský and Krůza2007; Konkol and Konopik Reference Konkol, Konopik, Habernal and Matoušek2013). Seven different named entity types are labeled in this corpus. The number of labels for each of these entity types for training, validation and test portions of the dataset is given in Table 2. For each word, the morphological analysis provided in the dataset was used. The fastText algorithm was used to obtain pretrained word embeddings of size 10 and 100 for Czech using the Czech version of Wikipedia (Bojanowski et al. Reference Bojanowski, Grave, Joulin and Mikolov2017).
The number of labels for each entity type in Czech and Finnish datasets
Hungarian: The Named Entity Corpus for Hungarian corpus that contains around 14,400 phrases tagged with entity labels were used (Szarvas et al. Reference Szarvas, Farkas and Kocsor2006). The corpus is labeled with the standard named entity tags. The training part contains 795 person names, 1,056 location names, 8,458 organization names and 1,327 miscellaneous names. The corpus originally contained only training and test parts, so validation and test sets were created by randomly selecting from the test part. The test set contains 100 person names, 125 location names, 1,055 organization names and 160 miscellaneous names. In the validation set, there are 87 person names, 113 location names, 1,020 organization names and 174 miscellaneous names. A statistical morphological analysis tool for Hungarian was used to process each word (Zsibrita et al. Reference Zsibrita, Vincze and Farkas2013) and its output was used as the input for morphological embeddings. For word embeddings, the fastText algorithm was used to obtain pretrained word embeddings of size 10 and 100 for Hungarian using the Hungarian version of Wikipedia (Bojanowski et al. Reference Bojanowski, Grave, Joulin and Mikolov2017).
Finnish: A labeled corpusFootnote 5 that was compiled from news articles in an online Finnish technology news site was used. The articles were published between 2014 and 2015. Extracting the morphological tags was done by a Finnish morphological analysis tool called Omorfi. Morphological disambiguation was done by FinnPos while creating the training and test sets (Silfverberg et al. Reference Silfverberg, Ruokolainen, Lindén and Kurimo2016). This corpus is labeled with five more named entity tags in addition to the standard set: ‘DATE’ for depicting date references, ‘EVENT’ for marking events, ‘PRO’ for marking products, ‘TIM’ for marking time expressions and ‘TIT’ for titles. The number of labels for each entity type is shown in Table 2. The fastText algorithm was used to obtain pretrained word embeddings of size 10 and 100 for Finnish using the Finnish version of Wikipedia (Bojanowski et al. Reference Bojanowski, Grave, Joulin and Mikolov2017).
Spanish: CoNLL 2002 Shared TaskFootnote 6 publishes a corpus tagged with NER and POS labels, which has clearly defined training, development and testing portions of the dataset. This dataset is widely used in NER related research for benchmarking. The POS tags were treated as the morphological analysis of the word as the POS tag contains the morphological information associated with the word if there is any. This corpus contains 6,278 person names, 6,981 location names, 10,490 organization names and 2,957 names of miscellaneous types. The fastText algorithm was used to obtain pretrained word embeddings of size 10 and 100 using the Spanish version of WikipediaFootnote 7 (Bojanowski et al. Reference Bojanowski, Grave, Joulin and Mikolov2017).
5.3 Results
This section presents the results of experiments performed to measure the impact of using morphological information for the NER task along with character-based embeddings. The experiments are conducted with the Turkish, Czech, Hungarian, Finnish and Spanish languages.
The experiments are performed with alternative embedding configurations, which are referred to with Setup followed by an integer as an identifier. The setups are as follows: (i) only pretrained word embeddings (Setup 1), (ii) only word and character embeddings (Setup 2), (iii) only word and a choice of one of the morphological embedding configurations (Setups 3–6) and (iv) word, character and a choice of one of the morphological embeddings (Setups 7–10).
Table 3 summarizes these results. A comparison of the basic model (Setup 1) with those that use morphological information (Setups 3–6) shows a performance increase when morphological information is used (ME(CHAR) and ME(WOR)).
The performance of the model using various embedding configurations for five languages (boldface values indicate the best values among models for the corresponding language)
In the case of Setup 4, an improvement is observed only for Turkish. Also, again for Turkish, using only the tags after the last derivational boundary (ME(WR_ADB)) is one of the most successful morphological configurations.
The performances of ME(CHAR) and ME(WOR) are comparable with the latter being slightly lower (except for Czech). The difference in the performance between Setups 5 and 6 may stem from the errors present in the morphological analyses. These errors are mostly due to unknown or misspelled words. In such cases, the analysis in the corpus usually defaults to the same nominal case. The higher performance of ME(CHAR) may be attributed to the ability to handle possibly faulty roots as morphological embedding captures more useful information in comparison to ME(WOR).
Another advantage of ME(CHAR) embeddings is its ability to capture the relationship between roots with the same prefix. For example, in the Finnish corpus, the frequencies of words with the same prefix often differ significantly based on their roots, such as in the cases of ‘allekirjoittaa’ (sign) versus ‘allekirjoittaja’ (signatory) and ‘Tampere’ (a city in Southern Finland) versus ‘Tamperelainen’ (of Tampere), where the occurrence of the former is much greater than the latter. The ME(CHAR) scheme also benefits from the common parts in related morphological tags. For instance, in Turkish, the tags ‘A3sg’ and ‘A3pl’ denote third person singular and third person plural, respectively, where the leading two characters ‘A3’ indicate third person agreement. The model can capture this information when the tags are represented in terms of characters. Therefore, ME(CHAR) is a better representation than either ME(WR) or ME(WOR).
Using only character embeddings in addition to the word embeddings (Setup 2) also improves the NER performance. Combining character embeddings with the ME(CHAR) and ME(WOR) models (Setups 9 and 10) outperforms all other setups (except Czech). For all the languages, the best performance is achieved with Setup 9 where all types of embeddings are employedFootnote 8. Although adding CE to Setup 1 (word embeddings) causes a large improvement, adding CE to Setup 5 (word embeddings and character embeddings of morphological part) provides a relatively small increase in performance (Setup 9). This may be the result of CE and ME(CHAR) both taking part in representing morphological information of words.
The results of these experiments show that an increase in NER performance is observed for all languages when either of CE or ME is included in the word representation. The best performance is achieved when both CE and ME are included in the word representation for all languages.
Table 4 shows a comparison of the proposed approach with the state-of-the-art results reported in literature. The CE+ME(CHAR) configuration (Setup 9) is used for comparison, since it yielded the best results. The models for each language were trained with higher number of parameters. The values for cell dimension of the sentence-level LSTM, character and morphological LSTM cell dimensions and character and morphological embedding dimensions were all set to 200. The word embeddings was set to 100. Other hyper-parameters and training related settings were unaltered. This work is the first one to report test results for Czech, Turkish, Hungarian, Finnish and Spanish. As such, comparisons are made using different studies for each language. For Turkish, a comparison with three different results are presented. The performance of Şeker and Eryiğit (Reference Şeker and Eryiğit2012) shown in the table was obtained using gazetteers. When such resources are not used, the performance drops to 89.55 per cent. Kuru et al. (Reference Demir and Özgür2016) did not employ any external data. Demir and Özgür (Reference Demir and Özgür2014) relied on hand-crafted features, however exploit externally trained word embeddings. For Finnish, the creators of the dataset are in the progressFootnote 9 of publishing a NER model. For Spanish, the result from Lample et al. (Reference Lample, Ballesteros, Subramanian, Kawakami and Dyer2016) is reported. A noteworthy observation is the increase in NER performance for Spanish even though its morphological characteristic differs from the other languages.
Comparison of results with state-of-the-art NER results for each language (boldface values indicate the best values among models for the corresponding language)
6 Conclusions
In this work, a state-of-the-art system for NER in MRLs was introduced. The impact of alternative combinations for embedding morphological tags that were examined revealed that augmenting word representations with morphological embeddings improves NER performance, which is further improved when combined with character-based word representations. Experiments with five languages, all MRLs except Spanish, were performed. The results obtained using this approach are the state-of-the-art for all of these languages. An ablation study to examine the impact of using morphological information revealed that the improved performance was similar across these languages.
Although extensive experiments with the interaction of morphological analysis based and surface form based representations were done, a correlation analysis was not conducted. A future work can give insight on the morphological representation quality of the character-based representation by inspecting the conditions of this correlation.
