2.1 Morphological analysis of Hebrew

In theoretical, descriptive, and computational work on Semitic languages (e.g. Glinert, Reference Glinert1989; Segal, Reference Segal2000; Buckwalter, Reference Buckwalter2002; Watson, Reference Watson2002), it is commonly assumed that a word may consist of the following elements:

• • Prefixes (or proclitics): conjunctions, prepositions, complementizers, and the definiteness marker (in a strict well-defined order).

• • Stem: a potentially polymorphemic part of the word, together with its POS tag and additional information about its template/pattern and root morphemes (see below).

• • Inflectional suffix: encoding inflectional features on verbs, adjectives, and nouns.

• • Pronominal suffix/enclitic: pronominal complements with verbs and prepositions, or possessive pronouns with nouns.

• • A feature indicating whether a noun, adjective, or numeral is a construct state or an absolute form.Footnote ¹

Except for the definiteness marker in Hebrew (see below), these word parts are simply concatenated in a strict order.Footnote ² For instance, the word wfnpgfnw (‘and that we met’, pronounced ve-she-nifgashnu)Footnote ³ is traditionally analyzed as follows:

Exmaple 1

• • Prefixes: the conjunction w (‘and’) and the complementizer f (‘that’)

• • Stem: the verb npgf (past tense of ‘meet’)

• • Inflectional suffix: nw (‘we’)

Some parts of this analysis are very robust and do not depend on the tagset or conventional notations. For instance, the identification of the prefixes w and f as separate segments, with their independent POS tags, is something that any linguistic analysis of Hebrew texts should assume. These morphemes are typical clitics: they have an independent syntactic role but are phonologically bound to the stem. Other decisions concerning the annotation scheme, especially concerning the status of the inflectional suffix, are sometimes less clear cut. For instance, in the verbal form npgfnw, whether to analyze the pronominal suffix nw as a separate word segment or as an inflectional part of the verbal stem may depend on the task at hand. In this work, we analyze inflectional suffixes as part of the stem, which is compatible with analysis of inflectional suffixes as part of the word in English and other languages.

Another non-trivial morphological phenomenon in Hebrew and other Semitic languages concerns the analysis of the templates and patterns for verbs, nouns, and adjectives. For instance, the meaning and inflectional features (tense, number, gender, person) of the verb npgf(nw) are identified using a set of morphological rules for the Hebrew verbal templates. These rules derive different verbs from one abstract root — a sequence of (usually three) consonants. In the case of the verb form npgf(nw), the root is pgf, and the template formation rule n + 〈root〉 (traditionally called ‘npel’) is responsible for forming the verbal stem npgf. Full morphological analysis of a Semitic word should include its root and template/pattern, but the usefulness of this morphological information may depend on the task at hand (see more on this point below).

Table 1.

Some usages of the Hebrew definiteness marker

One crucial problem for morphological disambiguation in Hebrew texts, which plays a special role in this paper, is the possible omission of the definite article. This prefix can be covert, which happens (only) when it appears together with one of three prepositional prefixes (b=‘in’, l = ‘to’ or k = ‘like/as’). The presence of a covert definiteness marker can often be recovered using the context. Some examples for the usage of the Hebrew definiteness marker are given in Table 1. Note that the ambiguity here is only due to the lack of vocalization in the modern Hebrew writing system, and ambiguous occurrences of words such as bbit have two different pronunciations (babayit or bebayit). Note further that in Hebrew (as in Arabic), modifying adjectives are marked for definiteness, and their marking should agree with the definiteness of the noun they modify. This facilitates the disambiguation of modified nouns that are orthographically ambiguous with regard to definiteness.

2.2 An annotation scheme for Hebrew POS tagging

From Example 1 above, it is clear that the morphological analysis of a Hebrew word may involve more than one element that has a syntactic role. In addition, the conjunction, the complementizer, and the verb may belong to different constituents in the syntactic structure, and may therefore need to have separate POS tags in any syntactic representation that contains the word wfnpgfnw. More generally, for Hebrew POS tagging we need to define the possible word segments and the POS tag that is assigned to any such word segment. By the term word segmentation, we henceforth refer to identifying the prefixes, the stem, and various suffixes of the word. By POS tag disambiguation, we mean the assignment of a proper POS tag to each of the identified word segments.

The task of segmentation and POS tag disambiguation is closely related to morphological analysis and disambiguation, but the two tasks are different at some respects. First, the internal analysis of the stem can be ignored for the sake of analyzing its POS tag. This is because the root and pattern/template morphemes are phonologically and syntactically bound, and the syntactic function of the stem is derived from them in a complex manner that need not concern applications such as syntactic parsing. Conversely, some syntactic distinctions that are relevant for POS tagging are not crucial for morphological disambiguation. For instance, the clitic f (‘that’) in Hebrew, like its parallel in English, is ambiguous between a complementizer and a relative pronoun. This ambiguity should be resolved by a syntactic parser or a POS tagger, but is not crucial for the internal morphological analysis of Hebrew words. Whether full morphological disambiguation is required in addition to POS tagging depends on the target application. For instance, full morphological disambiguation is important for machine translation and text-to-speech applications for Semitic languages. However, for syntactic parsing, chunking or information extraction, the POS tagging scheme that we describe below may often be sufficient.

For the purpose of our POS tagging scheme, we ignore part of the information that is found in the common morphological scheme shown in Section 2.1. The internal morphological structure of stems is not analyzed, and the POS tag that is assigned to a stem includes no information about its root, template/pattern, inflectional features, and suffixes. Among the suffixes, only pronominal complement suffixes on verbs and prepositions are identified as separate word segments. The construct state/absolute feature and the existence of a possessive suffix are identified using the POS tag assigned to the stem, and not as a separate segment or feature.

Some of these conventions are illustrated below by the segmentation and POS tagging of the word wfnpgfnw, as opposed to its traditional morphological analysis in Example 1.

Our segmentation and POS tagging conform with the annotation scheme used in the Hebrew treebank (Sima'an et al. 2001), as described below. Ignoring part of the morphological information in the analysis allows us to achieve reduced sparseness when training a POS tagging model on the small annotated corpus that is available for Hebrew, but may result in analyses that are not fully disambiguated morphologically. However, it is interesting to note that only 6.6% of the words that were tagged correctly by our system were not fully disambiguated. If one of the possible full analyses is picked randomly in such cases, then approximately 96.4% of the correct analyses can be expanded correctly to a full morphological analysis. Therefore, tagging using our scheme can be significantly helpful as a first stage in full morphological disambiguation. Having said that, in this paper, we concentrate on the tasks of word segmentation and POS tagging, and leave the study of full morphological disambiguation to further research.Footnote ⁴

2.3 A note on the POS tag annotation scheme for Arabic

Despite the similarity between the writing systems and morphology of Hebrew and Arabic, two differences should be mentioned. First, unlike the Hebrew definiteness marker (h), which can be covert, the definiteness marker al/l is always present in the Arabic writing system. Furthermore, the Arabic treebank (Maamouri et al. Reference Maamouri, Bies, Buckwalter and Mekki2004) does not separate the al/l prefix from the stem following it. This convention helps POS tagging in Arabic when compared with Hebrew. In contrast, however, pronominal suffixes of nouns and verbs are much more productive in Arabic than in Hebrew. In the Hebrew treebank, unlike the Arabic treebank, pronominal suffixes are scarce on nouns and virtually nonexistent on verbs (cf. Table A6 below). In this respect, POS tagging of nouns and verbs may often be easier than in Arabic.

4.1 Defining the input/output

As indicated in Section 1, word-level and segment-level tokenizations are the simplest options for defining the tokenization level for POS tagging of Hebrew and other Semitic languages. Before implementing these two alternatives and developing further variations, let us see how the choice between them directly affects the terminal and the nonterminal (output) symbols for POS tagging.

Word segments (WS)

Assuming a segmentation of the input words, the terminals of the tagger are segments, and the nonterminals are the usual POS tags. Note that in this case, information about how segments combine into words is not directly available for the disambiguator.

Table 2.

n-gram counts for each tokenization levelFootnote *.

* For each level of tokenization, the n-gram order actually used is marked in boldface.

Table 2 shows the number of different types (unigrams) found in the training corpus for the word and segment tokenization levels.

It should be stressed that word-level and segment-level tokenizations are two extremes, and the optimal level of tokenization that we propose for Hebrew POS tagging lies somewhere between them. However, the W/WS opposition is useful for the time being for introducing the key concepts of the proposed architecture.

4.2 The probabilistic framework

Let w^k ₁ be the input sentence, a sequence of words w ₁ . . . w _k. With word-level tokenization, the disambiguator aims at finding the analysis sequence a₁^k that has the highest probability given the sentence w^k ₁:

(1)

This is the standard formulation of probabilistic tagging for languages such as English.

Alternatively, with segment-level tokenization, the disambiguator aims at finding a combination of a segmentation sⁿ ₁ and a tagging tⁿ ₁ for sⁿ ₁, such that their joint probability with the given sentence, w^k ₁, is maximized:

(2)

where ANALYSES(w^k ₁) is the set of candidate analyses for the input sentence w^k ₁ (output by the morphological analyzer). Note that n can be different from k, and may vary for different segmentations. The original sentence can be uniquely recovered from the segmentation and the tagging. Since all 〈sⁿ ₁,tⁿ ₁〉 pairs that constitute the input to the disambiguator were derived from w^k ₁, we have P(w^k ₁|sⁿ ₁,tⁿ ₁)=1, and thus P(w^k ₁,sⁿ ₁,tⁿ ₁)=P(tⁿ ₁,sⁿ ₁). Therefore, (2) can be simplified as follows:

(3)

(1) and (3) can be represented in a unified formula that applies to both word-level and segment-level tokenizations:

(4)

In (4), eⁿ ₁ represents either a sequence of words or a sequence of segments, depending on the level of tokenization. The nonterminals Aⁿ ₁ are the respective analyses of these terminals—POS tag sequences or POS tags, respectively. The disambiguator aims at finding the most probable pair 〈terminal sequence, nonterminal sequence〉 for the given sentence, whereas in the case of word-tokenization, there is only one possible input terminal sequence for the sentence.

4.3 HMM probabilistic model

The actual probabilistic model used in this work for estimating P(eⁿ ₁,Aⁿ ₁) is based on HMMs. HMMs underly many successful POS taggers for different languages, for example DeRose (1988), Church (Reference Church1988), Cutting et al. (Reference Cutting, Kupiec, Pedersen and Sibun1992), Charniak et al. (Reference Charniak, Hendrickson, Jacobson and Perkowitz1993), Weischedel et al. (Reference Weischedel, Schwartz, Palmucci, Meteer and Ramshaw1993), and Merialdo (Reference Merialdo1994) for English, Dermatas and Kokkinakis (Reference Dermatas and Kokkinakis1995) for Dutch, English, French, German, Greek, Italian and Spanish, Brants (Reference Brants2000) for German and English, Hakkani-Tür et al. (2000) for Turkish, and many more.

For a k-th order Markov model (k=1 or 2), we rewrite (4) as:

(5)

For reasons of data sparseness, the instance models employed here work with k=2 for the segment-level tokenization, and with k=1 for the word-level tokenization. It is crucial at this point to observe that a Markov model at the word-level tokenization employs different statistics than a Markov model of the same order at the segment-level tokenization. Like in the case of English, the Hebrew segment tokenization working with a first-order Markov model conditions a single POS tag over a single preceding POS tag. In contrast, for the Hebrew word-level tokenization, a first-order Markov model will condition a sequence of POS tags for the current word on another sequence of POS tags for the preceding word. For example, consider again the phrase ‘bbit hgdwl’ from Table 1, tagged as bbit/IN-H-NN hgdwl/H-JJ. First-order Markov model will condition H-JJ on IN-H-NN (i.e. estimate P(H-JJ | IN-H-NN)).

For both models of HMM-based POS tagging, two kinds of probabilities need to be estimated: P(e _i|A _i) (the lexical model) and P(A _i|A _i−k,. . ., A _i−1) (the language model). Because the only manually POS tagged corpus that is available to us for training the HMM is relatively small (<4% of the Wall Street Journal WSJ portion of the Penn treebank), major effort must be dedicated to alleviating the sparseness problems that arise. For smoothing the language model probabilities over nonterminals, we employ the standard backoff smoothing method of Katz, (Reference Katz1987). Naturally, the relative frequency estimates of the lexical model suffer from more severe data sparseness than the estimates for the language model. On average, 31.3% of the test words in our setting did not appear in the training corpus. Next, we describe a novel method for smoothing the lexical probabilities using an untagged corpus.

4.4 Bootstrapping a better lexical model

We use an unsupervised training method for smoothing the lexical model acquired from the annotated data. For the sake of exposition, we assume word-level tokenization in this subsection. The method used for smoothing the segment-level tagger is very similar.

The smoothing of the lexical probability of a word w given an analysis a, that is, P(w|a) = P(w, a)/P(a), is accomplished by smoothing the joint probability P(w, a) only, that is, we do not smooth the relative frequency estimate of P(a).Footnote ⁷ For smoothing P(w, a), we use a linear interpolation of the relative frequency estimates from the annotated training corpus (denoted rf_tr(w,a)) together with estimates obtained by unsupervised estimation from a large unannotated corpus (denoted em_auto(w,a)):

(6)

where λ is an interpolation factor, experimentally set to 0.85.

Our unsupervised estimation method for obtaining em_auto(w,a) starts out with a naively smoothed POS tagger, and then tries to reestimate the lexical model probabilities of this POS tagger on an untagged corpus. In fact, this method can be viewed as a single iteration of the Baum–Welch (forward–backward) estimation algorithm (Baum Reference Baum1972) with minor differences. We apply this unsupervised method to an untagged corpus of 337K words.

The reestimation method starts out with a POS tagger that employs naively smoothed relative frequency estimates of the lexical model probabilities:

(7)

where f _tr(w) is the frequency of w in the training corpus and p ₀ is a constant (fixed at 10⁻¹⁰ in our experiments). We use the notation P _basic to denote the POS tagger that combines a Katz backoff smoothed language model together with the naively smoothed lexical model P _{LM 0}.

The unsupervised reestimation algorithm works in two steps. In the first step (expectation calculation), the model P _basic is employed in order to generate a distribution of alternative analyses (segment-tag sequences) for each of the sentences in the untagged corpus.Footnote ⁸ This way the untagged (incomplete) corpus is transformed into an ambiguously tagged (complete) corpus, containing for every sentence a distribution of analyses (i.e., segment-tag sequences), each weighted with its probability according to the model P _basic. Subsequently, in the second step (maximization), the ambiguously tagged, complete corpus is used to reestimate the lexical model probabilities utilizing relative frequency estimation (which in the case of a complete corpus is equivalent to maximum-likelihood estimation). Note that because this is an unsupervised training, the test set sentences may be added to the untagged corpus to ensure that the model assigns non-zero probabilities to the test set words.Footnote ⁹

This smoothing method can be seen as a variant of the well-known expectation-maximization (EM) method (Elworthy Reference Elworthy1994; Merialdo Reference Merialdo1994) for smoothing lexical probabilities using unlabeled data. In this approach, we initialize the model with relative frequency estimates from labeled data and reestimate the probabilities by running the EM algorithm on the unlabeled data. Our proposed method differs from this classical approach in the following points: First, the use of a weighting factor to control the influence of the unlabeled data, following (Nigam et al. Reference Nigam, Mccallum, Thrun and Mitchell2000). Second, we perform only one cycle of reestimation. This is justified by Merialdo's finding that additional iterations do not improve accuracy when starting with a relatively small tagged corpus. Finally, unlike standard EM that affects all model parameters, we smooth only the lexical probabilities, while using standard backoff methods to smooth language model probabilities. This is justified by the fact that sparseness is far more severe for lexical probabilities than for language model probabilities. To the best of our knowledge, this aspect of our method of lexical smoothing is novel.

5.1 From morphological analayses to candidate POS taggings

The set of possible morphological analyses was obtained from Segal's morphological analyzer (Segal's 2000). The analyzer's dictionary contains 17,544 base forms that can be inflected. After this dictionary was extended with the tagged training corpus, it recognizes 96.14% of the words in the test set. Segal's analyses follow the scheme described in Section 2.1. Translating them to the POS tagging scheme described in Section 2.2 involves three elements:

1. (1) Translating the POS of the stem to the treebank tagset. Since this tagset is more fine-grained than Segal's, the translation is one-to-many in some cases, thereby increasing the ambiguity. The treebank scheme uses different POS tags to represent some of the morphological features. For example, nouns in the absolute/construct state are represented as NN/NNT, respectively.

2. (2) Creating separate segments for the prefixes, along with their POS tag. This mapping can be ambiguous as well: according to the treebank tagset, the prefix f may be tagged either as a complementizer (COM) or as a relativizer (REL).Footnote ¹⁰

3. (3) Creating suffix segments for pronominal complement suffixes of verbs and prepositions.

Out of vocabulary (OOV) words (words that are not recognized by the morphological analyzer) are assumed to be proper nounsFootnote ¹¹ and the morphological analyzer proposes possible segmentations for the word on the basis of the recognition of possible prefixes. We enhanced this to allow analysis as ‘h’+common noun in case the OOV word begins with h. Furthermore, we applied a simple heuristic for reducing the number of possible segmentations for OOV words. The idea is that if two OOV words differ only in their (presumed) prefixes, it is more likely that they share the same stem and differ in their prefixes than that they have similar but different stems. For example, if the OOV word is wlalprd, ‘and to Alfred’, the letters w,l can be either prefixes or part of the stem. Now, if we see another OOV word malprd, ‘from Alfred’, we may assume with high probability that letters w,l are indeed prefixes, and eliminate segmentations in which these letters are part of the stem. The heuristic makes use of a collection of OOV words, extracted from a raw corpus containing 11 million words. The contribution of the OOV heuristic is analyzed in Section 6.6.

5.2 Implementing word-level and segment-level disambiguators

We used the SRI Language Modeling (SRILM) Toolkit (Stolcke Reference Stolcke2002) for constructing smoothed language models for tag sequences, and for finding the most probable tag sequence, we used Viterbi decoding (Viterbi, Reference Viterbi1967). We created from the small tagged corpus, a backoff language model for tag sequences, using Good-Turing discounting (Good, Reference Good1953) with Katz backoff (Katz, Reference Katz1987).

The construction of word-level (W) HMMs is well studied in the literature and is straightforward in SRILM. This is the standard HMM implementation that works with unambiguous input (a finite sequence of terminals). Clearly, this implementation is not directly suitable for modeling segment-level HMM taggers, since the disambiguator has to optimize not only the nonterminal sequence but also the terminal sequence (which corresponds to the segmentation).

For modeling the segment-level tagger, we simulate segment tagging by a word tagger, where the terminals are words and the nonterminals are analyses (tag sequences). As explained in Section 4.1, for a given word w and an analysis a=[t ₁ . . . t _p] for w that contains tags without the corresponding word segments, it is possible to extend a to a unique analysis ã=[(s ₁,t ₁) . . . (s _p,t _p)], where s ₁ . . . s _p are the respective segments. Therefore, at the sentence level, it is easy to move back and forth between segment-level analyses of the form (sⁿ ₁,tⁿ ₁) and word-level analyses of the form (w^k ₁,a₁^k). To simulate a segment tagger, we now only have to change the calculation of the probabilities P(a_i|a_i−2,a_i−1) and P(w _i|a_i), so that P(w^k ₁,a₁^k)=P(sⁿ ₁,tⁿ ₁) for each (w^k ₁,a₁^k) and the corresponding (sⁿ ₁, tⁿ ₁).

The probabilities are calculated as follows. First, we calculate segment-level probabilities: language model probabilities for tag sequences P _ws(t _i|t _i−2,t _i−1) and lexical probabilities for segment-tag pairs P _ws(s _i|t _i). The next step is to calculate P(a|a_i−2,a_i−1) and P(w _i|a_i):

• For P(a_i|a_i−2,a_i−1): Let a_i=[t ₁, . . ., t _p], a_i−1=[y ₁, . . ., y _q], and a_i−2=[x ₁, . . ., x _r]. Let u be the concatenation of a_i−2, a_i−1, and a_i:

  (8)

  
  where
  (9)

  
  Then,
  (10)

  
  Unlike standard n-gram backoff models that contain probabilities only for tag sequences found in the corpus, here we need to compute this probability for each possible trigram a_i−2a_i−1a_i, even if no subsequence of this trigram appears in the corpus. Thus, computing all these probabilities offline is inefficient, and will result in a very large model. To save run time and space, these trigram probabilities are derived on demand at run time; for each input sentence, based on the segment-level probabilties (P _ws), computed at training time.

• For P(w _i|a_i): Let ã_i=[(s ₁,t ₁) . . . (s _p,t _p)] be the full analysis, obtained from w _i and a_i. Then, P(w _i|a_i) is given by:

  (11)

  

It can be verified that multiplying the probabilities in (10) and (11) together gives the path probability according to the segment HMM model (see Equation 3).

Table 2 shows the average number of unigrams, bigrams, and trigrams in the training set for each tokenization level (this is an average over the five training sets in our cross-validation experiments, described in the next section). The relevant number for the actual n-gram size used for each level of tokenization appears in boldface. The table clearly shows the influence of the tokenization level on the data sparseness: word-level models suffer increased data sparseness in comparison with segment-level models. These figures support our choice to use bigrams for the word level and trigrams for segment level, since trigrams seem too sparse for the word level. This decision was also supported empirically, as we found that using trigrams does not improve the accuracy for the word-level tagger.

6.1 Experimental setting

Each architectural configuration was evaluated in five-fold cross-validated experiments. In a train/test split of the corpus, the training set includes 1,598 sentences on average, which amount to 28,738 words and 39,282 word segments. The test set includes 250 sentences.

Per segment measures

We also report the F _β=1 measure, defined as:

(12)

where segmentation (tagging) recall is defined as the length of the longest common subsequence (LCS) of the output (tagged) segment sequence (WSS) and the gold (tagged) WSS, divided by the length of the gold WSS. Similarly, precision is obtained by dividing the LCS length by the length of the output WSS.

For each parameter, the average over the five folds is reported, with the standard deviation in parentheses. We used two-tailed paired t test for testing the significance of the difference between the average results of different systems. The significance level (p value) is reported.

6.2 Baseline models

We tested two baseline taggers: one is based on choosing the most frequent analysis and the other is a POS tagger based on a morphological disambiguator developed by Segal's (2000), (see Section 7.1). The most frequent tagger operates at the word level and implements the following simple rule:

(13)

where rf_tr(a) is the relative frequency of a in the tagged corpus. The most frequent model was also tested on POS tag disambiguation alone, where the input is the correct sequence of segments. In this case, the above formula was applied per segment, rather than per word.

The accuracy of Segal's tagger on our data sets is another reference point. The training corpus was translated to Segal's scheme, and the results were translated back to the treebank's morphological format.Footnote ¹²

Table 3.

Baseline results

The baseline results are listed in Table 3. The tagging accuracy per word achieved by the most frequent method (82.71%) is relatively low, as compared with the 90% or higher accuracy achieved by similar methods for English (Charniak et al. Reference Charniak, Hendrickson, Jacobson and Perkowitz1993). Table 4 shows some examples for tagging errors. Since an English word corresponds to a word segment in Hebrew, it might be more appropriate to use the F _β=1 measure per segment for the comparison with English. The result (85.34%) is still considerably lower than the equivalent for English. This difference supports our conjecture that the task of POS tagging in Hebrew is relatively difficult, given the small annotated corpus we have. The results per segment obtained with the gold segmentation (88.43%) are closer to the results for English.

Table 4.

Some examples for tagging errors with the most frequent tagger.Footnote *

* The number of occurrences in the training corpus is given in parentheses. The word ‘snj’ does not appear in the training corpus, therefore the frequencies of ‘NN’ and ‘VB’ are compared.

The tagging accuracy per word of 85.22% obtained for Segal's tagger is distinctly incompatible with the 96.2% reported in Segal's (2000), presumably for the following reasons. First, the translation from Segal's annotation scheme to the treebank format is imperfect. Second, Segal's scheme is not flexible enough to describe every combination of segments found in Hebrew (and in the corpus). But, presumably, the most important reason for the difference is that Segal ensured that the right analyses for all the test words always appear in the dictionary. In practice, however, for about 4% of the test words, the right analysis does not appear in the dictionary, even after it is complemented with the training corpus.

Table 5.

Level of tokenization – experimental results

6.3 Evaluating the word-level and segment-level models

The first two lines in Table 5 detail the results obtained for both word (W) and word segment (WS) levels of tokenization. The tagging accuracy of the segment tagger is considerably better than that achieved by the word tagger (difference of 0.77% with significance level p=0.019). This is in spite of the fact that the segmentation achieved by the word tagger is a little better (and a segmentation error implies incorrect tagging). Our hypothesis is that:

The hypothesis about the relative data sparseness in the word-level model is supported by the number of once occurring terminals at each level: 8,582 at the word level versus 5,129 at the segment level.

6.4 Error analysis and discussion

Motivated by this hypothesis, we next consider what kind of word-level information is required for the segment-level tagger in order to do better in segmentation. Error analysis revealed a very common type of segmentation errors, which was found to be considerably more frequent in segment tagging than in word tagging. This kind of errors involves words such as bbit (‘in-the-house’/‘in-a-house’, cf. Table 1), where the definiteness marker h may be textually covert. The errors detected miss a covert h or wrongly add an unnecessary h to the analysis. Unlike other cases of segmentation ambiguity, which often just manifest lexical facts about spelling of Hebrew stems, this kind of ambiguity is productive: it occurs whenever the stem's POS allows definiteness, and is preceded by one of the prepositions b/k/l. In segment tagging, this type of error was found on average in 1.71% of the words (50% of the segmentation errors). In word tagging, it was found only in 1.37% of the words (42% of the segmentation errors).

Since in Hebrew, there should be agreement between the definiteness status of a noun and its related adjective, definiteness ambiguities of nouns can sometimes be resolved syntactically. For instance, reconsider the disambiguated cases in Table 1:

‘bbit hgdwl’ implies b-h-bit (‘in the big house’)

‘bbit gdwl’ implies b-bit (‘in a big house’)

In contrast, in many other cases, both analyses are syntactically valid, and the choice between them requires consideration of a wider context, or some world knowledge. For example, in the sentence hlknw lmsibh (‘we went to a/the party’), lmsibh can be analyzed either as l-msibh (indefinite, ‘to a party’) or as l-h-mbsibh (definite, ‘to the party’). Whether we prefer ‘the party’ or ‘a party,’ depends on the contextual information that is not available to the POS tagger.

Lexical statistics can provide valuable information in such situations, since some nouns are more common in their definite form, whereas other nouns are more common as indefinite. For example, consider the word lmmflh (‘to a/the government’), which can be segmented either as l-mmflh or as l-h-mmflh. The stem mmflh (‘government’) was found 25 times in the corpus, out of which only 2 occurrences were indefinite. This strong lexical evidence in favor of l-h-mmflh is completely missed by the segment-level tagger, in which segments are assumed to be unrelated. The lexical model of the word-level tagger better models this difference, since it does take into account the frequencies of l-mmflh and l-h-mmflh in measuring P(lmmflh|IN-NN) and P(lmmflh|IN-H-NN). However, since the word tagger considers lmmflh, hmmflh (‘the government’), and mmflh (‘a government’) as independent words, it still exploits only part of the potential lexical evidence about definiteness.

6.5 Evaluating the WS + h model

To better model such situations, we changed the segment-level model as follows. In definite words, the definiteness article h is treated as a manifestation of a morphological feature of the stem. Hence the definiteness marker's POS tag (H) is prefixed to the POS tag of the stem. We refer by WS + h to the resulting model that uses this assumption, which is rather standard in theoretical linguistic studies of Hebrew (Wintner Reference Wintner2000; Danon Reference Danon2001). The WS + h model can be viewed as an intermediate level of tokenization, between segment-level and word-level tokenizations. In Table 6, we demonstrate the different analyses that are obtained by the three models of tokenization.

Table 6.

Representation of l-h-mmflh in each level of tokenization

As shown in Table 5, the WS + h model shows remarkable improvement in segmentation (0.50%, p=0.0018) compared with the initial segment-level model (WS). As expected, the frequency of segmentation errors that involve covert definiteness (h) dropped from 1.71% of the test words to 1.22%. The adjusted segment tagger also outperforms the word-level tagger in segmentation (by 0.31%, p=0.049). Tagging improved as well, although the significance level is low (0.32%, p=0.081). According to these results, tokenization as in the WS+h model is preferable to both plain-segment and plain-word tokenizations.

Table 7.

WS + h model: contribution of bootstrapping and the OOV heuristic

6.6 The contribution of the individual components

Having determined the optimal level of tokenization, we would now like to measure the contribution of individual components to the overall accuracy of our best model. We focus on the smoothing of the lexical model using untagged data (see Section 4.4), and the heuristic for reducing the number of possible segmentations for OOV words (see Section 5.1). The results are shown in Table 7. The first model uses a baseline lexical model, based on maximum-likelihood estimation from the tagged corpus (the model is similar to LM ₀ defined in Section 4.4, except that the level of tokenization here is word segments, rather than words). The results show that bootstrapping (second line) improves tagging by 1.53% and segmentation by 0.67%. The OOV heuristic (third line) further improves the segmentation accuracy by 0.29% and tagging by 0.1%. The improvements are all statistically significant (p<0.05).