1. Introduction
Statistical and neural parsing techniques are widely known to be surprisingly powerful and compelling, in the sense that they are easily trained and evaluated if the syntactically annotated treebank is available. Their results, however, can be difficult to understand because parsing models by the state-of-the-art statistical and neural parsers are automatically induced from the treebank. Treebanks, collection of parsed and annotated corpora, constitute an essential resource for natural language processing in any given language (Abeillé Reference Abeillé2003). The automatic syntactic analysis of sentences directly benefits syntactically annotated corpora. Currently, most of the state-of-the-art parsers use annotated syntactic information such as constituent structure or dependency relations to train probabilistic models. Several annotated phrase structure Korean treebanks have been created over the last two decades, such as the KAIST treebank (Choi et al. Reference Choi, Han, Han and Kwon1994), the Penn Korean treebank (Han et al. Reference Han, Han, Ko, Palmer and Yi2002) and the Sejong treebank (released in 2011).Footnote a Although there has been a shared task of constituent parsing for Korean using the KAIST treebank as part of the shared task on Statistical Parsing of Morphologically Rich Languages (Seddah et al. Reference Seddah, Tsarfaty, Kübler, Candito, Choi, Farkas, Foster, Goenaga, Gojenola Galletebeitia, Goldberg, Green, Habash, Kuhlmann, Maier, Nivre, Przepiórkowski, Roth, Seeker, Versley, Vincze, Woliński, Wróblewska and de la Clergerie2013; Seddah, Kübler, and Tsarfaty Reference Seddah, Kübler and Tsarfaty2014), studies on constituent parsing for Korean using these treebanks have hardly been performed. For instance, there are just a few previous works that used these treebanks such as Chung et al. (Reference Chung, Post and Gildea2010), in which they are using the Penn treebank, and Choi et al. (Reference Choi, Park and Choi2012) and Park et al. (Reference Park, Hong and Cha2016) where they are using the Sejong treebank.
1.1 Various phrase structures in Korean treebanks
Morphological analysis for Korean has been based on an eojeol, which has been considered as a basic unit of segmentation in Korean, delimited by blank spaces in the sentence. Given that Korean is an agglutinative language, joining content and functional morphemes of eojeols is very productive. The number of types for possible combined eojeols can be increased exponentially. For example, unggaro (‘Ungaro’) is a content morpheme (a proper noun) and a postposition -ga (nominative) is a functional morpheme. They form together a single eojeol unggaro-ga (‘Ungaro + nom’). For convenience sake, we add a hyphen (-) at the beginning of functional morphemes, such as -ga for nom, to distinguish between content and functional morphemes. The nominative case markers -ga or -i may vary depending on the previous letter – vowel or consonant. A predicate naseo-eoss-da also consists of the content morpheme naseo (‘become’) and its functional morphemes, -eoss (‘past’) and -da (‘ind’), respectively.
-
(1)
Meanwhile, different granularity levels for word segmentation have been independently proposed as different basic segmentation units for Korean treebanks based on the combination of content and functional morphemes. For example, the Sejong treebank (Figure 1(a)), the most widely used corpus for Korean, uses the eojeol as itself for the basic unit of analysis. That is, no tokenisation scheme in the treebank is proposed. The token in the Sejong treebank, therefore, may contain punctuation marks and other symbols along with words. The Sejong treebank explicitly utilises the binary structure using Chomsky normal form.
Different Korean treebanks using the different segmentation granularity levels. (a) The Sejong treebank structure (b) The Penn Korean treebank-like structure (c) The KAIST treebank-like structure.
There are, however, treebanks which use an English-like tokenisation and multifurcating trees. Words in these corpora are already preprocessed, for example, the Penn Korean treebank (Han et al. Reference Han, Han, Ko, Palmer and Yi2002), in which punctuation marks are separated from words (Figure 1(b)). Moreover, the Penn Korean treebank particularly annotates these null elements in the sentence. Null elements in the Penn Korean treebank represent empty arguments (*pro*), traces (*T*), empty operators (*op*) and predicate deletions (*?*) (Han, Han, and Ko Reference Han, Han and Ko2001). We also note that there are ver. 1.0 and 2.0 in the Penn Korean treebank, in which their characteristics of the corpus are completely different. The text of the former is from military manuals and the latter is from the Korean Newswire corpus which is a collection of Korean Press Agency news articles.
Government and Binding (GB) theory (Chomsky Reference Chomsky1981, Reference Chomsky1982, Reference Chomsky1986) for Korean often proposed a syntactic analysis, in which the entire sentence depends on verbal endings. The KAIST treebank (Choi et al. Reference Choi, Han, Han and Kwon1994), the earliest Korean treebank introduced this type of analysis (Figure 1(c)). In the KAIST treebank, functional morphemes are explicitly separated from their content morphemes. The SPMRL 2013 and 2014 shared tasks used the modified version of the KAIST treebank for Korean parsing (Seddah et al. Reference Seddah, Tsarfaty, Kübler, Candito, Choi, Farkas, Foster, Goenaga, Gojenola Galletebeitia, Goldberg, Green, Habash, Kuhlmann, Maier, Nivre, Przepiórkowski, Roth, Seeker, Versley, Vincze, Woliński, Wróblewska and de la Clergerie2013, Reference Seddah, Kübler and Tsarfaty2014). Whereas the original KAIST treebank explicitly separates postpositions and verbal endings from the phrase as in Figure 1(c), the modified KAIST treebank uses an eojeol-based tokenisation scheme as in the Sejong treebank.
Figure 1 shows different Korean treebanks using the different morphological segmentation granularity of the word in their phrase structures. Whereas there are studies on merging heterogeneous treebanks to improve parsing results such as Stymne et al. (Reference Stymne, Lhoneux, Smith and Nivre2018) for dependency parsing, these three treebanks for Korean would still be difficult to be interchangeable because of the differences of their lexical and syntactic annotations. It is thus not easy to combine them together in order to obtain the bigger treebank.
1.2 Previous work on constituent parsing for Korean
One of the earliest works on statistical constituent parsing for Korean is Hermjakob (Reference Hermjakob2000) where he used 1187 treebank sentences taken from the Korean newspaper Chosun Ilbo (‘Korea Daily News’). He implemented a shift-reduce parser based on his earlier work (Hermjakob and Mooney Reference Hermjakob and Mooney1997) with syntactic and semantic features supported by background knowledge such as a time expression or a locative marker which are generally not covered by part-of-speech information. The average sentence length in the treebank was 21.0 words. He presented evaluation results with training sentences ranging from 32 to 1024, and the remaining 163 sentences of the treebank for testing. He got a relatively high-performance result with 91.0% (precision) and 89.8% (recall). Currently, the Chosun treebank is not available.
Sarkar and Han (Reference Sarkar and Han2002) trained lexicalised tree adjoining grammars (LTAGs) from 4,653 sentences (91% of the corpus) of the early version of the Penn Korean treebank (ver. 1.0). LTAG grammars are extracted using LexTract (Xia et al. Reference Xia, Han, Palmer and Joshi2000). The statistical parser was implemented based on Sarkar (Reference Sarkar2002) and used three probabilistic models: one probability for suppertagging of the elementary tree with a lexical anchor (Bangalore and Joshi Reference Bangalore and Joshi1999) and two probabilities for substitution and adjunction operations where elementary trees can be combined with. The parser was then tested on 425 sentences (9%), and they obtained a 75.7% F
$_1$
score.
Chung et al. (Reference Chung, Post and Gildea2010) used all 5010 sentences in the Penn Korean treebank (ver. 2.0) using 10-fold cross-validation for experiments. They obtained results up to a 89.56% F
$_1$
score with the parsing techniques described in Post and Gildea (Reference Post and Gildea2009). They used an implementation of a CKY PCFG parser by Johnson (Reference Johnson1998). Their treebank grammars are extracted by the Berkeley parser (Petrov and Klein Reference Petrov and Klein2007) using various factors including non-terminal granularity and null elements.
After Park (Reference Park2006) introduced treebank grammars extracted from the early version of the Sejong treebank and its usability in the LTAG parser, Choi et al. (Reference Choi, Park and Choi2012) used the Sejong treebank for constituent parsing. They proposed an entity-based preprocessing where certain set of morphemes are merged as a single entity, instead of a morpheme-based parsing as depicted in Figure 2(c), and improved parsing accuracy. The Sejong treebank does not distinguish special symbols from punctuation marks in the eojeol, both of which are annotated as a punctuation mark. During the entity-based preprocessing, entities, such as numbers with the decimal point (3.14), email addresses (name@email.com), web address (http://www.web.info), dates (25/9/2017), can be presented as a single token, while punctuation marks are still separated from the eojeol. Choi et al. (Reference Choi, Park and Choi2012) introduced the Stanford parser that uses a generalised chart-based CKY algorithm (Klein and Manning Reference Klein and Manning2003), the Bikel parser (Bikel Reference Bikel2004a) that introduces lexicalised rules based on the Collins parser (Collins Reference Collins1999) and the Berkeley parser (Petrov and Klein Reference Petrov and Klein2007) that proposes the probabilistic context-free grammar with latent annotations. They reported a 73.92% F
$_1$
score with the Stanford parser, 75.41% with the Bikel parser and 77.29% with the Berkeley parser based on a 10-fold cross-validation. These parsers are the state-of-the-art statistical parsers at the moment, in which parsers can elaborate context-free grammars by refining annotations of non-terminal symbols in the treebank. On the same extension, Park et al. (Reference Park, Hong and Cha2016) continued to work on constituent parsing using the Sejong treebank. They obtained parsing results up to a 85.12% F
$_1$
score (with gold part-of-speech (POS) labels) using the PCFG-LA Berkeley parser by correcting syntactic and POS annotation errors in the Sejong treebank. Note that Choi et al. (Reference Choi, Park and Choi2012) and Park et al. (Reference Park, Hong and Cha2016) used the black box experiments using the default option of each parser.
An example of the Sejong treebank. (a) The original Sejong treebank annotation. (b) The original Sejong treebank. (c) The Sejong treebank converted to the morpheme-based treebank for constituent parsing.
Seddah et al. (Reference Seddah, Tsarfaty, Kübler, Candito, Choi, Farkas, Foster, Goenaga, Gojenola Galletebeitia, Goldberg, Green, Habash, Kuhlmann, Maier, Nivre, Przepiórkowski, Roth, Seeker, Versley, Vincze, Woliński, Wróblewska and de la Clergerie2013) and Seddah et al. (Reference Seddah, Kübler and Tsarfaty2014) presented the modified KAIST treebank at the SPMRL 2013 and 2014 shared tasks for Korean parsing. One of the participants obtained results up to 83.81% F
$_1$
score for Korean by adding the reranking step to the Berkeley parser (Björkelund et al. Reference Björkelund, Çetinoğlu, Farkas, Mueller and Seeker2013, Reference Björkelund, Çetinoğlu, Faleńska, Farkas, Mueller, Seeker and Szántó2014), in which reranking has been proven effective to refine parsing results (Collins Reference Collins2000; Charniak and Johnson Reference Charniak and Johnson2005; Collins and Koo Reference Collins and Koo2015). While this result for Korean outperformed the earlier neural parsers such as Durrett and Klein (Reference Durrett and Klein2015), more recent neural parsers reported up to 88.96% (Coavoux and Crabbé Reference Coavoux and Crabbé2016; Kitaev and Klein Reference Kitaev and Klein2018; Kitaev, Cao, and Klein Reference Kitaev, Cao and Klein2019) from the same data set.
Hermjakob (Reference Hermjakob2000), Sarkar and Han (Reference Sarkar and Han2002) and the SPMRL KAIST data set utilised an English-like tokenisation scheme for the input, in which punctuation marks are separated from words.Footnote b However, many downstream applications for Korean language processing are based on morphological segmentation granularity, in which all morphemes are separated: POS tagging (Jung, Lee, and Hwang Reference Jung, Lee and Hwang2018; Reference Kitaev, Cao and Klein2019), machine translation (Park et al. Reference Park, Hong and Cha2016, Reference Park, Dugast, Hong, Shin and Cha2017), etc. The main reason they introduce this morpheme-based granularity is to avoid the data sparse problem because longer segmentation granularity such as eojeols can yield a number of possible words exponentially, and it results in the inevitable data sparsity problem. Therefore, current phrase structure parsing for Korean has also utilised morphemes-based granularity as a de facto standard (Chung et al. Reference Chung, Post and Gildea2010; Choi et al. Reference Choi, Park and Choi2012; Park et al. Reference Park, Hong and Cha2016).
Whereas the interpretation of a direct comparison of results between the different granularity levels has been considered as being difficult because their representation levels are different (e.g., the number of terminal lexical tokens is different), we empirically found that morpheme-based granularity outperforms the English-like tokenisation scheme for Korean language processing applications including constituent parsing if they are fairly converted into the same granularity level.
1.3 Goal of the paper
The recent development of constituent parsing has focused on attention-enhanced neural models (Vinyals et al. Reference Vinyals, Kaiser, Koo, Petrov, Sutskever, Hinton, Cortes, Lawrence, Lee, Sugiyama and Garnett2015; Kuncoro et al. Reference Kuncoro, Ballesteros, Kong, Dyer, Neubig and Smith2017; Kitaev and Klein Reference Kitaev and Klein2018; Kitaev et al. Reference Kitaev, Cao and Klein2019) and domain-independent agnostic models (Vinyals et al. Reference Vinyals, Kaiser, Koo, Petrov, Sutskever, Hinton, Cortes, Lawrence, Lee, Sugiyama and Garnett2015; Fried, Kitaev, and Klein Reference Fried, Kitaev and Klein2019). Reranking (Charniak and Johnson Reference Charniak and Johnson2005) has been tested for neural constituent parsing (Fried, Stern, and Klein Reference Fried, Stern and Klein2017), and dynamic oracles for dependency parsing (Goldberg, Sartorio, and Satta Reference Goldberg, Sartorio and Satta2014) have also been applied to constituent parsing (Fernández-González and Gómez-Rodríguez Reference Fernández-González and Gómez-Rodríguez2018; Fried and Klein Reference Fried and Klein2018). Using the Berkeley parser (Petrov and Klein Reference Petrov and Klein2007), the Trance parser (Watanabe and Sumita Reference Watanabe and Sumita2015) and Berkeley neural parser (Kitaev and Klein Reference Kitaev and Klein2018; Kitaev et al. Reference Kitaev, Cao and Klein2019), we train and evaluate the phrase structure Korean Sejong treebank. As shown in Dakota and Kübler (Reference Dakota and Kübler2017) for other languages and treebank, we define the best practices for replicability in constituent parsing for Korean, including correct comparison to future work by proposing the standard corpus division for the Sejong treebank. The other main contribution of this paper is to present two important factors on constituent parsing for Korean: detailed qualitative and quantitative parsing error analyses (Sections 3 and 4, respectively) using the Sejong treebank.
2. Experiments and results
2.1 Parsing evaluation methodology
In previous work on parsing for Korean, either phrase structure or dependency parsing, Park et al. (Reference Park, Kawahara, Kurohashi and Choi2013) proposed the 80-10-10 corpus split for training, development and evaluation, whereas others often used cross-validation (Choi et al. Reference Choi, Park and Choi2012; Oh and Cha Reference Oh and Cha2013). Table 1 from Park et al. (Reference Park, Hong and Cha2016) presents the current baseline parsing results using phrase structure grammars by the PCFG-LA Berkeley parser. Park et al. (Reference Park, Hong and Cha2016) performed 5-fold and 10-fold cross-validation as well as corpus split evaluation methodologies for comparison purposes. Reported results are improved compared to Choi et al. (Reference Choi, Park and Choi2012) because syntactic and POS tagging errors have been corrected in the Sejong treebank. Since results between different evaluation methods are relatively small, we propose to divide the corpus into a 80-10-10 ratio for the Sejong treebank.
Phrase structure parsing result comparison by Park et al. (Reference Park, Hong and Cha2016)
2.2 Data and parsing systems
The current available version of the Sejong treebank (released in 2011) contains about 41,000 sentences in 15 files. The KAIST treebank used in SPMRL and the Penn Korean treebank in Chung et al. (Reference Chung, Post and Gildea2010) also used an 80-10-10 ratio for each file. While Choi and Palmer (Reference Choi and Palmer2011) proposed the similar approach for dependency parsing, using the Sejong treebank, their version of the Sejong treebank is not currently available and they did not specify the exact file split. The contents of the Sejong treebank represent a variety of sources, newswire text, magazine articles on various subjects and topics, several book excerpts, and crawled texts from the internet. Therefore, we divide up each treebank file into an 80-10-10 ratio as in the previous work and merges them together into a training, a development and a test data set for constituent parsing. Currently, it contains 33,224/4144/4144 sentences for training, development and test, respectively. Table 2 shows the proposed number of sentences in each file in the Sejong treebank. If appropriate, the proposed split can be served as a 90-10 ratio without a development data set.
The proposed number of sentences from the Sejong treebank to consist train/dev/test partitions
For our experiments, we make use of the morphological analysis annotations in the treebank and focus on the parsing task, in which we directly use morpheme-separated information in the treebank instead of end-to-end parsing. For example, a relative clause verb segyejeokin (‘world-class’) is analysed and segmented into morphemes segye (noun), -jeok (derivational suffix), -i (copular) and -n (relative marker) in the treebank. POS tags are integrated as leaf nodes alongside their morphemes. We preprocess the treebank data by converting POS tags into immediate non-terminal labels as in the previous work by Choi et al. (Reference Choi, Park and Choi2012). Consequently, the originally binary tree-structured Sejong treebank may contain multifurcating branches for the word such as
$[_{\textsc{VNP-MOD}}$
$[_{\textsc{NNG}}$
segye ]
$[_{\textsc{XSN}}$
jeok ]
$[_{\textsc{VCP}}$
i ]
$[_{\textsc{ETM}}$
n ]], and a word segyejeogin is converted into segye
$\sqcup$
jeog
$\sqcup$
i
$\sqcup$
n where
$\sqcup$
represents a blank space for the input of parsing systems.
Figure 2 shows an example of the Sejong treebank for the sentence in (1): Figure 2(a) for the original treebank annotation, Figure 2(b) for the tree-style original treebank and Figure 2(c) for the converted multifurcating treebank for constituent parsing.
We use the Berkeley parser (Petrov and Klein Reference Petrov and Klein2007)Footnote c for statistical parsing, and the Trance parser (Watanabe and Sumita Reference Watanabe and Sumita2015)Footnote d and the Berkeley neural parser (Kitaev et al. Reference Kitaev, Cao and Klein2019)Footnote e for neural parsing with their default options. We invite you to see their github pages for the detailed option values. The Berkeley parser uses the probabilistic context-free grammar with latent annotations (PCFG-LA, Matsuzaki, Miyao, and Tsujii Reference Matsuzaki, Miyao and Tsujii2005) framework and performs series of split and merge cycles of non-terminal nodes to maximise the likelihood of a treebank. The Trance parser is a transition-based shift-reduce parser using compositional vector grammars described in Socher et al. (Reference Socher, Bauer, Manning and Ng2013). The Berkeley neural parser uses a sentence encoder based on self-attention for a chart decoder (Kitaev and Klein Reference Kitaev and Klein2018), and later extends to unsupervised pre-training using Bidirectional Encoder Representations from Transformers (BERT) (Devlin et al. Reference Devlin, Chang, Lee and Toutanova2019). The Berkeley neural parser reported that it outperformed the best result of SPMRL 2014 by Björkelund et al. (Reference Björkelund, Çetinoğlu, Faleńska, Farkas, Mueller, Seeker and Szántó2014) for Korean. These parsers show competent results across a variety of languages, and they are easy to train on any treebank. These factors make the current experiments more objective and reproducible. The Trance parser can take word embeddings (Mikolov et al. Reference Mikolov, Chen, Corrado and Dean2013a; Pennington, Socher, and Manning Reference Pennington, Socher and Manning2014; Bojanowski et al. Reference Bojanowski, Grave, Joulin and Mikolov2017), which has shown to benefit many natural language processing applications including the constituent parsing system such as Socher et al. (Reference Socher, Bauer, Manning and Ng2013). We collect text from the Sejong raw corpus, the Korean Wikipedia (accessed on August 20, 2017), and all published articles during 2016 in The Hankyoreh (‘The Korean Nation’) for a 100-dimensional skip-gram model (Mikolov et al. Reference Mikolov, Sutskever, Chen, Corrado, Dean, Burges, Bottou, Welling, Ghahramani and Weinberger2013b) to build a word embedding vector. We use bert-base-multilingual-uncased for pre-training to combine with the Berkeley neural parser. For experiments in this paper and error analyses in Sections 3 and 4, we use Penn-like preprocessing for the treebank, especially by removing function labels in the phrase label such as -sbj or -obj as described in Bikel (Reference Bikel2004b). We also convert the POS label np (pronoun) into prop to avoid confusing the noun phrase (also np) in the treebank.
Overall parsing results using pcfg-la berkeley (Petrov and Klein Reference Petrov and Klein2007), trance (Watanabe and Sumita Reference Watanabe and Sumita2015) and neural berkeley (Kitaev and Klein Reference Kitaev and Klein2018; Kitaev et al. Reference Kitaev, Cao and Klein2019)
2.3 Results
Table 3 shows overall parsing results using the Berkeley parser (Petrov and Klein Reference Petrov and Klein2007), the Trance parser (Watanabe and Sumita Reference Watanabe and Sumita2015) and the Berkeley neural parser (Kitaev and Klein Reference Kitaev and Klein2018; Kitaev et al. Reference Kitaev, Cao and Klein2019). We evaluate the parser accuracy with the standard F
$_1$
metric from EVALB.Footnote f Park et al. (Reference Park, Hong and Cha2016) reported that results from the Berkeley parser improve up to 85.12% using the gold POS tags. It is also feasible to use a 90-10 ratio without development data for the Berkeley parser. While the original Berkeley parser paper (Petrov et al. Reference Petrov, Barrett, Thibaux and Klein2006) proposed several runs of training because of the EM algorithm which can find locally maximum likelihood parameters, we empirically found that each run of training gives the same results. Therefore, we can use train and dev data together for the single run of training using the PCFG-LA Berkeley parser. While we obtain a 84.31% F
$_1$
score without gold POS labels and 85.47% with gold POS labels for a 90-10 corpus split, the difference of results are still relatively small.Footnote g We obtain 87.24% using the Trance parser.Footnote h We train and evaluate the Berkeley neural parser (Kitaev and Klein Reference Kitaev and Klein2018; Kitaev et al. Reference Kitaev, Cao and Klein2019). It obtains up to 88.67% and 89.18% f
$_1$
scores without and with BERT, respectively. The result by the Berkeley neural parser using BERT (Kitaev et al. Reference Kitaev, Cao and Klein2019) significantly outperforms other previous parsing results using the Sejong treebank.Footnote i The observed p-values compared to results by the Berkeley parser (Petrov and Klein Reference Petrov and Klein2007) and the Trance parser (Watanabe and Sumita Reference Watanabe and Sumita2015) are much less than 5%.Footnote j
Chung et al. (Reference Chung, Post and Gildea2010) reported that including function labels such as -sbj or -obj can improve parsing results for the Penn Korean treebank. While such function labels appear only at np in the Penn treebank, any phrase labels can have function labels in the Sejong treebank. For example, miggeuleoji (‘slide’, a verb), -m (noun derivational suffix), and -do (‘also’) is annotated as vp-sbj in the Sejong treebank because a noun derived from a verb is annotated as a vp and its function in the sentence is a subject. Such diversities of phrase labels with function labels in the Sejong treebank can give a biased distribution of phrase labels, and we hardly improve parsing results using such labels. Practically, the result with function labels is decreased by between 1% and 2% for all statistical and neural parsers. Table 4 shows parsing results using the Berkeley parser (Petrov and Klein Reference Petrov and Klein2007) per iteration with the numbers of non-terminal nodes and CFG rules. As the number of non-terminal nodes increases, the number of the rules of CFG increases exponentially, thereby leading to the increase in the complexity in parsing. Taking this result into consideration, this study can make a judgement that using such a phrase label would bring a certain degree of limitations to improving the practical effect of their results. The same is true of the French treebank as described in Park (Reference Park2018). The improvement in the results of Chung et al. (Reference Chung, Post and Gildea2010) using the Penn Korean treebank might be due to the restrictions corresponding to the relatively small number of sentences in the treebank.
Parsing results using pcfg-la berkeley per iteration with the numbers of non-terminal nodes and CFG rules
3. Qualitative parsing error analysis
In this section, we find six different types of common errors in the course of qualitatively analysing the parsing data entirely based on our linguistic intuitions.Footnote k
-
1. error from dropping (case) markers,
-
2. error in recognising arguments of a predicate,
-
3. error in modifying verb phrases,
-
4. error in modifying the noun phrase (NP) with relative clauses,
-
5. error in identifying the category, and
-
6. error in identifying lexical properties like polyseme and homonym.
Most of these types of errors are mainly due to the linguistic characteristics of the Korean language such as head-final word order, and case, pronoun and conjunction dropping behaviours caused by the morphological richness. The Korean language has SOV (subject-object-verb) word order, which means that the head in a sentence occurs in a final position and that a sentence can, at least, take two consecutive NPs and that the distance between the main verb and its subject can also be much bigger than we might expect. These linguistic properties of the Korean language significantly influence the following types of errors in parsing. We detail these errors, and the following figures present parsed results and correct parse trees.
Error from dropping (case) markers: the first noun phrase undonglyang (‘amount of exercise’) should have a nominative case marker -i as in undonglyang-i.
3.1 Error from dropping (case) markers
Among the linguistic features of the Korean language stated above, the three linguistic features of dropped case marker, pronoun and coordinating conjunction trigger the errors in appropriately modifying NPs. In the Korean language, case markers attached to NPs can be omitted. They would be both markers such as nominative and accusative cases in the subject or object positions, genitive case of nouns preceded by the other nouns, and the postpostional markers attached to nouns denoting place, direction or time, etc. The genitive case marker, of course, might be found wherever two consecutive NPs occur in a sentence. The following example in (2) (as described in Figure 3) lies in the first case, the drop of a nominative case marker in subject position. The first NP undonglyang (‘amount of exercise’) should have a nominative case marker -i as in undonglyang-i, but it was omitted. This resulted in a parsing error. That is, it leaves the possibility in which the predicate manh-go can easily form one constituent with the immediate following element masaji (‘massage’), in particular because this predicate is connected to the coordinate conjunction -go and it usually tends to make a constituent with the following NP masaji, which, in fact, functions without a genitive marker as a modifier of the NP hyogwa (‘effect’). After all, it results in an erroneous analysis, where the NP masaji does not modify the immediate NP hyogwa. If the drop of the nominative case marker did not happen, the subject NP would be easily combined with the constituent manh-ko immediately followed by.
-
(2)
3.2 Error in identifying arguments of a predicate
Error in identifying the arguments of a predicate – a subject or sometimes an object – can be found in two situations, as shown in Table 5. One is the absence of subject in an embedded clause and the other is a rather long distance between the verb and its subject. First of all, the Korean language typically is a pro-drop language in which some pronouns may be omitted when they are contextually inferable. This property causes errors in identifying the subject when an embedded or matrix verb looks for its subject. Sometimes, the object itself can also lead to the same errors. Thus, when the pronoun in subject position is omitted, particularly in an embedded sentence, the verb in a subordinate clause can wrongly recognise the subject of a main clause as its own subject, since the subject in a subordinate clause has a relatively high possibility of being deleted if it has the same referent to that in the main clause. In addition to the pro-drop property, the Korean language displays an SOV word order, which means that the head occurs in the final position of the sentence and the distance between the main verb and its subject may be long. This linguistic property also leads to errors. Sometimes the pro-drop subject and the long distance between it and the verb combine to mislead the parsing.
Errors from argument dropping of a predicate: incorrect match of subject or object in embedded clause by matrix verb
The following sentence (3) (Figure 4) consists of two clauses, one of which has an embedded clause without a subject pronoun angyeong-eul sseosseul ttae (‘eyeglass-acc wear’). This environment causes error in recognising its subject. In this clause, the verb sseosseul (‘wear’) wrongly identified the subject of the main clause hangug sobija bohowon-i (‘Korea Consumer Agency-nom) as its own subject. The subject in the subordinate clause is not presented but, nonetheless, we can easily understand from the context that the entity who does the action of wearing eyeglasses would be people in general. That is, no one would comprehend that the actor wearing them must be the subject of a main clause, hangug sobija bohowon.
-
(3)
Error in identifying arguments of a predicate: the verb sseosseul (‘wear’) wrongly identified the subject of the main clause hangug sobija bohowon-i (‘Korea Consumer Agency-nom) as its own subject.
3.3 Error in modifying VP
When adverbs modify verb phrases (VPs), parsing errors become more likely. Especially, under the circumstance where the main verb in a sentence containing more than one embedded clause is modified by adverbial expressions denoting time and location, the distance between them can be longer than that of the verb in the embedded clause. A situation such as a long dependency between two related elements can presumably lead to the errors in modifying VP. In particular, when the adverbial expressions occur in the initial position of a sentence, we can quickly imagine two possible alternatives to verb modification: to modify the closest verb or the main verb in the final position. That is, sometimes, the adverbial expressions can modify the embedded verb occurring in the closer position than the main verb happening in the final position of the whole sentence. Sometimes, the adverbs of sentences without long dependency relation can also wrongly modify the verb in the final position by crossing beyond the element necessary to be correctly modified from them. For instance, the adverbial expression in the following sentence 82 nyeon-eneun (‘82 year-in’) should modify the immediately following verb yeolly-eoss-deon (‘held when’), but it leads to an error by modifying the main verb in the final position of the sentence. See the example in (4) (Figure 5).
-
(4)
Error in modifying VP: 82 nyeon-eneun (‘82 year-in’) should modify the immediately following verb yeolly-eoss-deon (‘held when’).
3.4 Error in modifying the NP with RC
NPs can typically be modified by relative clauses, but that can give rise to errors. When a relative clause modifies a NP containing more than one NP, the relative clause can modify inappropriate nouns. For instance, it only modifies the first noun instead of the second or last one, generating errors by forming a wrong constituent. The following sentence has a relative clause deowi-leul sighi-neun (‘heat-acc cool off-ajt’), and this modifies the head noun with two NPs yeoleum-yong (‘use for summer’). This situation produces an error in wrongly modifying the first head noun, forming a wrong constituent because the second NP in head nouns of a relative clause (RC) is regarded as another constituent and only the first NP is modified by the relative clause. See the example in (5) (Figure 6).
-
(5)
Error in modifying the noun phrase with RC: deowi-leul sighi-neun (‘heat-acc cool off-ajt’) should modify the head noun with two noun phrases yeoleum-yong (‘use for summer’).
3.5 Error in identifying category
As stated at the beginning of this section, the Korean language features a case marker drop. The contracted form of bound morphemes can also trigger errors in identifying phrasal categories in certain phrasal levels. The phrase anbang-seo in the following sentence (6) has the contracted form seo of bound morpheme e-seo. If we recover the source sentence of the following sentence, the postpositional marker e-seo (or seo, loc) can be attached to the noun anbang (‘main room’), resulting in forming a prepositional phrase denoting a location. Unfortunately, this contracted form seo has two meanings in this context: as a postpositional marker denoting a location or as a coordinating conjunction for connecting two events. In the sentence below, the bound morpheme seo has to be used as the former, but it was parsed as the latter, ending up with an error in recognising the wrong phrasal category for the sentence. See the example in (6) (Figure 7). We limit this type of error only to the case where detecting phrase boundaries is correct and the labelling of syntactic tags is not correct.
-
(6)
Error in identifying category: the bound morpheme seo has to be used as a postpositional marker denoting a location.
Error by lexical properties like polysemy and homonym: isang, should be interpreted as a meaning of ‘more than’.
3.6 Error by lexical properties like polysemy and homonym
Some lexical items with multiple meanings or with the same spelling but different meanings also lead to parsing errors. In the following example, we can find isang, which can be used as a polysemous word. This lexical word can be interpreted simultaneously as meaning ‘ideal’ as well as ‘more than’. In the former, this word can be a new single phrasal category by combining with the immediately preceding word, whereas in the latter, it denotes the amount of the preceding noun and functions as a quantifier. This lexical property of polysemy yields to errors in forming a proper constituent. In the context of the sentence in (7) (Figure 8), this word was used as the latter.
-
(7)
3.7 Discussion on qualitative error analysis
We explored qualitative error analysis by focusing on phrase boundary detection errors which yield incorrect boundaries and labels. The morphological richness and even SOV word order can also significantly influence the types of errors in parsing. We believe that current qualitative analysis might be estimated to cover over 90% of the parsed sentences with errors as measured by thoroughly examining manually the parsing results with our linguistic intuitions. There are also some minor errors such as advp advp errors for the adverbial phrase, in which a sentence is parsed either
$[_{\textsc{s}}$
$[_{\textsc{advp}}$
$[_{\textsc{advp}}$
adverb]
$[_{\textsc{advp}}$
adverb]]
$[_{\textsc{s}}$
sentence]] or
$[_{\textsc{s}}$
$[_{\textsc{advp}}$
adverb]
$[_{\textsc{s}}$
$[_{\textsc{advp}}$
adverb]
$[_{\textsc{s}}$
sentence]]] because the Sejong treebank annotates the sequence of the adverbial phrases in both ways inconsistently. It would be, however, difficult to quantify parsing errors that result from qualitative classification, for example, the number of errors from dropping case markers where we cannot count the number of dropping case markers. In addition, a distinction between errors might remain vague, for example, an error in Section 4.2 can be classified as an error in recognising arguments of a predicate and an error in modifying the NP with relative clauses. Therefore, we develop an error typology based on phrase labels in the next section to quantify parsing errors, from which we may systematically elaborate to improve future parsing results.
4. Quantitative parsing error analysis
4.1 Methodology
Kummerfeld et al. (Reference Kummerfeld, Hall, Curran and Klein2012, Reference Kummerfeld, Tse, Curran and Klein2013) presented an approach to quantify constituent parsing errors for English and Chinese. We extend this method to Korean by adding advp attachment and word boundary errors. The following errors are considered from parsing results. We use parsing example for quantitative error analysis using the PCFG-LA Berkeley parser.
4.2 Quantitative error typology
NP attachment NP attachment errors are the most frequent errors in constituent parsing for Korean. They consist in mistakes and inconsistencies in recognising arguments of a predicate. While the first NP 40dae-buteo (‘from 40s’) in (8) (Figure 9) is an argument of the subordinate clause insaeng-ui soetoegi-ga sijakdoe-n-da-neun (‘has started the decline of life’), it is dominated by the main clause and it results in an erroneous analysis.
-
(8)
NP attachment error: the noun phrase 40daebuteo (‘from 40s’) should be the argument of the subordinate clause insaengui soetoegiga sijakdoendaneun (‘has started the decline of life’).
VP attachment When two (or several) verbal phrases (VPs) appear consecutively and a parser identifies them as complex predicates, it results in (9a) (Figure 10). However, it leads to an error by modifying the main verb in the final position of the sentence. A VP dallaji-go-i-ss-da-neun (‘being differentiated’) is the predicate of the subordinate clause and the entire subordinate clause should modify the main verb bunseogdeul-i-da (‘be analyzed’) in (10) and Figure 10.
-
(9)
VP attachment error: the verbal phrase dallaji-go-i-ss-da-neun (‘being differentiated’) should be the predicate of the subordinate clause. It also shows a different label error where the root non-terminal symbol should be a vnp.
AdvP attachment When adverbial phrases (AdvPs, annotated as an ap in the Sejong treebank) modify verb phrases, parsing errors become more likely. Adverbs and adverbial phrases modifying whole sentences often appear at the beginning of the sentence. While an AdvP gyeolko (‘never’) in (10) (Figure 11) modifies the whole sentence malhaji anhneun (‘not say’) at the beginning (of vnp), the parser analyses it as modifying the VP of the subordinate clause ttabunhadago (‘being bored’).
-
(10)
Adverbial phrases attachment error: the AdvP gyeolko (‘never’) should modify the whole sentence malhaji anhneun (‘not say).
Clause attachment Clause attachment errors are similar to VP attachment errors. We differentiate them based on their phrase labels, either s or vp. The predicate of the clausal argument jeogeojyeoss-gi (‘reducement’) in (11) (Figure 12) is analysed as a complex predicate, in which it is dominated by VNP (noun verbal phrase) also because an AdvP useon (‘first’) is also wrongly parsed to modify the entire vnp.
-
(11)
Clause attachment error: the predicate of the clausal argument jeogeojyeoss-gi (‘being decreased’) should be analysed as a complex predicate.
Word boundary Compound words in Korean can be written alternatively with or without a blank such as yeoreobunyaeseo or yeoreo bunyaeseo (‘in several fields’). Since yeoreo (‘many’) in (12) (Figure 13) also appears independently several times in the corpus, the parser analyses a single word yeoreobunyaeseo as yeoreo bunyaeseo because the input is morpheme-separated (yeoreo bunya eseo).
-
(12)
Quantitative error analysis results. The number of errors based on parsing results w/o function labels: np for np attachment errors, wb for word boundary errors, vp for vp attachment errors, dl for different label errors, cl for clause attachment errors, advp for advp (adverbial phrase) attachment errors and un for undefined errors
Word boundary error: the parser should analyse a single word yeoreobunyaeseo.
Different label Lastly, the parser can attribute wrong phrase labels such as s instead of vnp for sentences as in (9) (Figure 10) because there are sentences in which their starting symbols are a verbal phrase label (vp or vnp).
4.3 Discussion on quantitative analysis
Table 6 shows the numbers of errors in the test data set based on the different parser settings. Since the neural parsing system outperforms the statistical parsing systems for all types of errors, it would be easy to conclude that the recent neural parser performs better for the any sentence pattern qualitatively and quantitatively throughout the treebank. While Kummerfeld et al. (Reference Kummerfeld, Hall, Curran and Klein2012) presented different error quantities for English depending on parsers, the ratio for error types are consistent for Korean regardless of the parser type. Therefore, presented parsers show the similar tendency in the ratio of the frequency for error types. The best neural system yields up to less than 27.31% error rate for all attachment errors compared to the baseline pcfg-la berkeley setting. It especially reduces the error rate by 36.71% for phrase labelling. We have 35,565 NPs, 28,304 VPs, 2976 AdvPs and 4010 S (clause) fragments in the test data set. It suggests that about 10% of NP (np), AdvP (advp) and S (cl) labels are involved in parsing errors while less than 5% for VP (vp) labels for the PCFG-LA Berkeley parser. For example, we observe 11.0951% of NP fragments as NP attachment errors which would be closely related with errors from dropping case markers (Section 3.1) and from identifying arguments of a predicate (Section 3.2) in qualitative parsing error analysis. Therefore, reductions in NP errors by correctly identifying arguments of the predicate, for example, would result in large gains on F-scores. As Kummerfeld et al. (Reference Kummerfeld, Hall, Curran and Klein2012) remarked, some of the undefined errors could be categorised in errors described above; undefined errors can also contain a long tail of other errors, which we do not define. About 20% of parsing errors are undefined, and they require more fine-grained definition for quantitative error types, and we leave this for future work.
5. Technical discussion on parsing error analyses
As mentioned in earlier sections, this study mainly keeps close track of both the qualitative and quantitative analyses in order to find a good strategy for improving the results of constituent parsing for Korean. This study expects that this would lead to better results in the sense that two analyses can complement each other, by looking at their results with a multilateral perspective.
First of all, we could confirm that both analyses produce well-grounded results than we anticipated at the outset. The qualitative error analysis reliably allows us to detect phrase boundary errors, decisively influencing the error types in parsing triggered by morphological richness and SOV word order. The quantitative analysis tends to provide stable results regardless of the parser types.
The qualitative analysis, on the other hand, is rather sensitive to the methods of annotating the sequence of the phrases. This study suggests an error typology based on phrase labels as an alternative strategy to compensate for this difficulty, which is a quantitative analysis to make it possible to quantify parsing errors. The quantitative analysis, however, is likely to yield undefined parsing errors, thus requiring more refined definition for quantitative error types. As can easily be noticed by many researchers, it would be great to effectively use a new method that takes the interaction of the current qualitative and quantitative analyses into consideration. We have some difficulty in further looking at the errors in qualitative analysis by comparing them with the recent neural parsers similarly to that of the quantitative analysis in two different aspects.
One reason lies in that it is unlikely to be easy to keep consistent track of errors in both the two analyses because the same type of error can occur more than one time within the same or part of the sentence in the qualitative analysis, unlike that in the quantitative analysis which is in a deterministic way. Several errors, sometimes, might appear within the same sentence. In addition, the qualitative analysis in this study has been performed by thoroughly investigating manually the parsing results with our linguistic intuitions with an expert knowledge in the fields of Korean syntax. It is hard to find a more good method to perform the work automatically than ours, inevitably leaving this for future work. This analysis needs more researching in the field of constituent parsing for Korean in the near future. The new attempts, therefore, to complement the shortcomings of these two analyses would face a new challenge to overcome, as stating above.
6. Conclusion
Treebank grammars used to be readily understandable even in the various annotation schemes, such as markovisation and head lexicalisation (Charniak Reference Charniak1996; Johnson Reference Johnson1998; Klein and Manning Reference Klein and Manning2001; Post and Gildea Reference Post and Gildea2009). Latent annotations in the Berkeley parser (Petrov et al. Reference Petrov, Barrett, Thibaux and Klein2006; Petrov and Klein Reference Petrov and Klein2007), on the other hand, made it difficult for humans to interpret treebank grammars. Whereas there are efforts to understand recent neural language models (Kulmizev et al. Reference Kulmizev, Ravishankar, Abdou and Nivre2020), the neural parsers, which estimate the parameters of the parsing model, came to be more difficult for humans to understand. Using the statistical and the neural parsers, however, this study has primarily tried at finding and identifying the concrete typology of parsing errors and additionally attempted to check how skillfully these parsers deal with sentences, focusing on the tendency of their distributional ratio. The most frequently occurring errors, no matter whether qualitative or quantitative analysis, concern the attachment problems mostly for the sentences with long distance dependency, in which the parser fails to find the correct argument of the predicate. Park et al. (Reference Park, Kawahara, Kurohashi and Choi2013) proposed a lexicalised probabilistic model with case frames aggregated from predicate-argument structures, and they improved the dependency parsing results for Korean. We expect to get some benefit from such information as case frames for constituent parsing, and we leave this for future work.
The main goal of the paper is to discuss widespread complicated issues on constituent parsing for Korean such as preprocessing of the treebank and error analysis, thereby improving parsing results. As a result of these works, we have properly defined where to train and evaluate the Sejong treebank, detailed constituent parsing errors, and have improved parsing results. We obtained a 89.18% F
$_1$
score using the neural parser, and it outperformed other previous work for Korean using the Sejong treebank.
Over the last few years, we have worked on many extrinsic factors for constituent parsing to improve parsing results for Korean: Choi et al. (Reference Choi, Park and Choi2012) transformed the treebank into entity level for a certain number of morphemes, and Park et al. (Reference Park, Hong and Cha2016) and this paper adapted the recent neural parser for Korean parsing. Our future work will focus on a parsing scheme itself to adapt linguistic properties, such as the morphological richness and SOV word order, which significantly influence the parsing results for Korean.
