2.1 Root-pattern morphology

Amharic is characterized by the root-pattern nonconcatenative morphology. A root, also called a radical, is a sequence of commonly three consonants which carry the basic lexical meaning. A pattern, also called a vocalic element, consists of vowels which are inserted (intercalated) between the consonants of the root. Stems are, therefore, formed by intercalating the vowels between the root consonants. Finally, prefixes and suffixes are added either to complete the stem as a word or to form another stem.

For example, the Amharic root sbr Footnote ¹ means ‘break’. If we insert the pattern ä-ä among the radicals, we obtain the stem säbbär-. Attaching the suffix -ä gives säbbärä ‘he broke’ which is the first form of the verb (third person masculine singular in past tense as in other Semitic languages) (Bender et al. Reference Bender, Bowen, Cooper and Ferguson1976). Using the same pattern, but without geminating the second consonant, and attaching the suffix -a, the process nominal säbära is produced which means ‘breaking’. The pattern ä-a combined with the suffix -i makes agent nouns, as in säbari ‘one who breaks’. The same stem with the suffix -a, gives us the participle säbara ‘broken’. Intercalating the vowel ä between the second and third consonant results in a jussive stem sbär-. Attaching the prefix mä- to it forms an infinitive verb mäsbär ‘to break’ (Bender et al. Reference Bender, Bowen, Cooper and Ferguson1976). The imperfective stem is derived from the root by intercalating the vowel ä between the first and the second consonants.

2.2 Derivational and inflectional morphology

There are five parts of speech in Amharic: adjectives, nouns, verbs, adverbs, and prepositions and conjunctions. Prepositions and conjunctions are totally unproductive. Adverbs are few in number and are less productive. They are not inflected but some adverbs can be derived from adjectives, for instance, kIfuña ‘severely or seriously’ is derived from the adjective kIfu ‘wicked’ by suffixing -ña.

Nouns are derived from other basic nouns, adjectives, stems, roots, and the infinitive form of a verb by affixation and intercalation. For example, from the noun lIǧǧ ‘child’ another noun lIǧnät ‘childhood’; from the adjective däg ‘generous’ the noun dägInnät ‘generosity’; from the stem sInf, the noun sInfInna ‘laziness’; from root k'ld, the noun k'äld ‘joke’; from infinitive verb mäsbär ‘to break’ the noun mäsbäriya ‘an instrument used for breaking’ can be derived. Case, number, definiteness, and gender marker affixes inflect nouns. Table 1 presents, as an example, the genitive case markers for nouns.

Table 1.

Genitive case markers

Adjectives are derived from nouns and stems by adding a prefix or a suffix, and they can also be derived from verbal roots by intercalating patterns. For example, it is possible to derive dIngayama ‘stony’ from the noun dIngay ‘stone’ and zIngu ‘forgetful’ from the stem zIng, by suffixation; sänäf ‘lazy’ can be derived from the root snf by intercalation. Adjectives can also be formed through compounding. For instance, hodäsäfi ‘tolerant, patient’, is derived by compounding the noun hod ‘stomach’ and the adjective säfi ‘wide’. Like nouns, adjectives are inflected for gender, number, and case (Yimam Reference Yimam2007).

Unlike the other word categories such as nouns and adjectives, the derivation of verbs from other parts of speech is not common. The conversion of a root to a basic verb stem requires both intercalation and affixation. For instance, from the root gdl ‘kill’ we obtain the perfective verb stem gäddäl- by intercalating the pattern ä_ä. From this perfective stem, it is possible to derive a passive (tägäddäl-) and a causative stem (asgäddäl-) using prefixes tä- and as-, respectively. Other verb forms are also derived from roots in a similar fashion. Verbs are inflected for person, gender, number, aspect, tense, and mood (Yimam Reference Yimam2007). Table 2 shows, as an example, the inflection of perfective and imperfective verbs for person, gender, and number. Other elements like negative markers and prepositions also inflect verbs in Amharic. A single Amharic word may, therefore, represent a complete English sentence. For instance, amt’Iččewalähu is equivalent to the sentence ‘I have brought it’. A more detailed description of Amharic morphology can be found in (Tachbelie Reference Tachbelie2010).

Table 2.

Inflection for person, gender, and number

From the above brief description, it can be seen that Amharic is a morphologically rich language. It is this feature that makes the development of language models for Amharic challenging.

3.1 Corpus preparation

To train language models, a text corpus (called ATC_48k hereafter) consisting of 48,090 sentences and 1,542,697 tokens has been prepared. The electronic text has been obtained from the ethiozenaFootnote ² archive which contains written newscast. Since the target application domain is speech recognition and since we wanted to merge the text corpus with the one prepared by Abate, Menzel and Tafila (Reference Abate, Menzel and Tafila2005) for speech recognition, the text has been normalized accordingly. The normalization task includes:

• • Expansion of abbreviations and contractions. The most common contractions are Ethiopian personal names, titles, and names formed through compounding. For example, the personal name gäbrämaryam is commonly written as gä/maryam, the title t’äk'lay ministär usually contracted as t’ä/ministär or sometimes as t’ä/mi.

• • Textual transcription of numbers. Telephone numbers, dates, times, etc. written in numbers have been transcribed.

• • Removal of punctuation marks.

• • Removal of foreign words, particularly English words provided that their removal does not affect the grammatical structure of a sentence.

After normalization, the ATC_48k corpus was merged with the text database prepared by Abate et al. (Reference Abate, Menzel and Tafila2005) from the same domain. The combined text corpus (hereafter called ATC_120k) was used in the experiments and consists of 120,262 sentences with 2,348,150 tokens and 211,120 word types. Table 3 presents the frequency distribution of words in the ATC_120k corpus.

Table 3.

Word frequency distribution

It closely follows Zipf's law with a few very common words, a middle number of medium frequency words, and very many low frequency words (Manning and Schütze Reference Manning and Schütze1999), which implies that the data is sparse. As can be seen from Table 3, more than 50% (121,285) of the words occur only once (hapax legomena) in the corpus. On the other hand, only five words appear with a frequency of above 10,000. These words are function words such as wIsT ‘in’.

We compared the frequency distribution of Amharic words with other languages, namely English and German. To make a fair comparison, we normalized the size of the available corpora to the smallest one, Negra which consists of 20,602 German sentences. The same number of sentences has been selected for English and Amharic from the Wall Street Journal (WSJ) and the ATC_120k corpora, respectively. Figure 1 depicts the frequency distribution of words for the three languages. As can be clearly seen, the number of hapax legomena for Amharic is much bigger than for German and English.

Fig. 1.

Word frequency distribution of Amharic, English, and German.

This issue attributes to the morphological richness of Amharic, which is revealed in a very high-vocabulary growth rate as presented in Figure 2.

Fig. 2.

Vocabular growth rate of Amharic, English, and German.

We have also computed the distribution of word lengths in terms of the pronunciation units.Footnote ³ As can be seen from Figure 3, there are Amharic words with up to 24 pronunciation units long but the maximum word lengths observed in English and German are 18 and 20, respectively. The highest proportion of Amharic words are 8–12 pronunciation units long while it falls between 4 and 8 for English and German.

Fig. 3.

Word length distribution of Amharic, English, and German.

We can understand, therefore, that the use of words as lexical and language model units for morphologically rich languages like Amharic imposes a severe data sparseness problem, high-vocabulary growth rate, high-OOV word rate, computational load of the pattern matching component that handles concatenation of lexemes (words) into whole hypothesized utterances. These issues motivate the use of subword lexical and language model units for Amharic.

3.2 Word segmentation

Developing a subword language model requires use of a word parser which splits word forms into its constituent morphemes. There have been a number of attempts to develop morphological analyzers for Amharic based on different approaches (Bayou Reference Bayou2000; Alemayehu and Willett Reference Alemayehu and Willett2002; Bayu Reference Bayu2002; Fissaha and Haller Reference Fissaha and Haller2003; Adafre Reference Adafre2005; Amsalu and Gibbon Reference Amsalu and Gibbon2005; Argaw and Asker Reference Argaw and Asker2007; Gasser Reference Gasser2011). However, most of the systems were either not suited for our purpose or have not been publicly available when needed. Moreover, morphologically annotated data for Amharic does not yet exist. Therefore, we adopted Morfessor (Creutz and Lagus Reference Creutz and Lagus2005), a language-independent unsupervised morphology learning tool, to produce a morphologically segmented corpus.

The Morfessor baseline model that we have used in our experiment is based on the minimum description length (MDL)Footnote ⁴ criterion. Based on the MDL principle, the Morfessor baseline learns a lexicon of morphemes which is concise and produces a compact representation for the words in the corpus (Creutz and Lagus Reference Creutz and Lagus2005). Details about Morfessor and how it works can be found in (Creutz and Lagus Reference Creutz and Lagus2005).

A word type list of the ATC_120k corpus has been used as input to Morfessor. Morfessor learns the morpheme segmentation from the input data and presents as output a word and its constituent morphemes (called statistical morphemes hereafter).

3.2.1 Evaluation of the Word Segmentation

The segmentation performance can be evaluated by comparing morpheme boundaries proposed by the algorithm with a linguistic gold standard. Creutz (Reference Creutz2006) indicated that such an evaluation is straightforward and intuitive provided that an adequate gold standard exists. Since, to our knowledge, there is no gold standard segmentation available for Amharic,Footnote ⁵ we manually prepared one for about 1,000 words. The words have been taken from the output segmentation (type-based segmentation) systematically. That means after taking the first word, every fiftieth word has been taken and the correct segmentation for it has been provided. Table 4 shows examples for the output of Morfessor and that of our manual segmentation. We can see the errors in Morfessor's segmentation.

Table 4.

Morfessor segmented data

However, similar to Creutz (Reference Creutz2006), we considered Amharic stems as a correct segmentation although they should have been analyzed into root and pattern. Morfessor's segmentation of words (type-based segmentation) has been compared with our manual segmentation and precision, recall and f-measure (see Table 5) have been calculated using the evaluation scripts of Hutmegs package (Creutz and Lind Reference Creutz and Lind2004). The f-measures for Amharic are higher than the one reported by Creutz (Reference Creutz2006) for Arabic (41.7%) using the same version of Morfessor but on a data set consisting of 17,000 distinct words. This might be due to the amount of data used for evaluation and the way the gold standards have been prepared. The Arabic data set is very big compared to the one used to evaluate the Amharic segmentation (only 1,000 word types). While the Arabic gold standard has been generated automatically based on a lexicon of Egyptian Colloquial Arabic, the Amharic one was prepared manually which would result in a more accurate segmentation compared to the automatic method.

Table 5.

Evaluation results of the segmentation

3.3 The language models

The words in our corpus (ATC_120k) have been replaced automatically by their constituent morphemes. The resulting morpheme-segmented corpus (called ATC_120k_ Seg, hereafter) consists of 4,035,656 tokens and 15,923 morpheme types. This corpus is divided into three parts: training set, development set, and test set with proportions of 80:10:10. The splitting is only for the language-modeling experiment. Perplexities, log probabilities, and OOV rate are calculated on the development set and test set. The development set is used to find the optimal options of language modeling (parameter tuning), for example, the best smoothing technique. The best performing model is then evaluated on the test set. However, word recognition accuracies and OOV rates reported in Section 6 (speech recognition experiments) are not based on the development and test sets of this corpus. They are calculated based on the 5k development set which is part of the read speech corpus (Abate et al. Reference Abate, Menzel and Tafila2005) used in our investigation.

Then, statistical morpheme-based n-gram models of order 2–5 were trained using the SRI Language Modeling Toolkit (SRILM) (Stolcke Reference Stolcke2002). The effect of different smoothing techniques (Good-Turing, Absolute Discounting, Witten–Bell, Natural Discounting, Modified and Unmodified Kneser–NeyFootnote ⁶) on the quality of a language model were then studied. The best results (in terms of perplexity) obtained for each smoothing technique are presented in Table 6. As can be seen from Table 6, the best performing model is a pentagram model with unmodified Kneser–Ney smoothing. This result is in line with the findings of Chen and Goodman (Reference Chen and Goodman1998), that Kneser–Ney and its variations outperform other smoothing techniques.

Table 6.

Perplexity of statistical morpheme-based language models

Probability estimates of different n-gram order have been interpolated for Witten–Bell, Absolute Discounting, and Modified Kneser–Ney smoothing techniques. Table 7 shows the best results for each smoothing technique. Interpolating n-gram probability estimates of a specific order n with lower order estimates sometimes yield better models. This is because, especially for low counts, lower order distributions provide valuable information about the correct amount to discount (Chen and Goodman Reference Chen and Goodman1998). Our experiment verified this fact. A pentagram model with Modified Kneser–Ney smoothing and interpolation of n-gram probability estimates has a perplexity of 101.38. For the other smoothing techniques an increase in perplexity has been observed.

Table 7.

Perplexity results with interpolation

The best performing model had a perplexity of 102.59 on the test set.

As indicated by Stolcke (Stolcke Reference Stolcke2002), discarding unknown words or treating them as a special ‘unknown wordFootnote ⁷’ (unk) token affects the quality of language models. A language model in which unknown words are treated as unk assigns a nonzero probability to OOV words by virtue of the mapping of OOVs to unk. This might improve the predictive power of the language model. Thus, unknown morphemes have been mapped to a special ‘unknown morpheme’ token for the best model from Table 7 but instead of the expected improvement an increase in perplexity (to 102.26) was observed. This might indicate that the OOV problem is already handled by the use of morphemes as units of the language model.

4.1 Comparison of word-based and statistical morpheme-based language models

Since test sets have quite different token counts for word- and morpheme-based language models, the perplexity values of these two kinds of language models are not directly comparable. Even so, a comparison can be made based on the probabilities the models assign to the test sets, where a model with a higher probability is considered a better one. To avoid underflow problems, log probabilities are commonly used. Table 9 depicts the log probabilities (calculated from the language model development set which is a 10% of the available data and is different from the training set and test set) of the best statistical morpheme- or word-based models as well as the correspondingFootnote ⁸ word-based or statistical morpheme-based models.

Table 9.

Log probabilities

As can be seen in Table 9, word-based models have a higher log probability and, therefore, should be considered superior ones although their perplexity is higher than that of the morpheme-based ones. On the other hand, subword-based language models offer the benefit of reducing the OOV words by 99%. This is a great achievement, as the OOV words problem is severe in morphologically rich languages in general, and Amharic in particular.

Our findings are somewhat contradictory so far. While perplexity measures favor the morpheme-based approach, the probability-based evaluation indicates an advantage for the word-based account. Therefore, a comparison within the framework of a target application is required to find out which model is better suited to improve the performance on the particular task. We used speech recognition for that purpose.

5.1 Manual word segmentation

Words in Amharic can be decomposed into root, pattern, and one or more affix morphemes, which are combined by intercalation and concatenation. Since Morfessor is not able to deal with the nonconcatenative feature, we manually segmented 72,428 word types found in a corpus of 21,338 sentences (419,660 tokens). By doing so, we aimed at obtaining an optimistic estimate of what a fully automatic procedure could achieve at best if it were available. In contrast to the mere segmentation provided by Morfessor, we complemented the manual segmentation with an annotation of the root and the pattern whenever appropriate.

We do not claim the manual segmentation to be comprehensive. Since a word type list has been used, there is only one entry even in case of polysemy or homonymy. For example, the word t’Iru might be an adjective which means ‘good’ or it might be an imperative verb which has a meaning ‘call’ (for second person plural). Consequently, the word has two different segmentations. Nevertheless, we provided only one segmentation based on the most frequent meaning of the word in our text. The geminated and nongeminated word forms, which might have distinct meanings and consequently different segmentations, have also been treated in the same manner as the polysemous or homonymous ones. Because the transcription system does not indicate the gemination of consonants, a systematic ambiguity remains. For instance, ?at’ägäbu can be treated as an adverb which means ‘next to him’ or as a verb with a meaning ‘they made somebody else full or they satisfied (somebody else)’ based on the gemination of the consonant g. This word could have, therefore, been segmented in two different ways: [?at’ägäb + u] if it is an adverb or [?a + t’gb + ää + u] if it is a verb.

Another point is related to the plural forms of words borrowed from GeezFootnote ⁹ which are formed by epenthesis and vocalic changes. For example, the plural form of the noun kokäb - ‘star’ is käwakIbt. There are also some Amharic words whose plural forms are formed through partial reduplication. For instance, the plural form of the adjective tIllIk’ - ‘big’ is tIlallIk’. It would have been easier to analyze such words as, for example, käwakIbt = kokäb + NPl Footnote ¹⁰ or tIlallIk’ = tIllIk’ + AdjPl Footnote ¹¹. However, we need an analysis that gives the morphemes themselves. Thus, such words have not been segmented since it is difficult to identify their morphemes. Nevertheless, we believe that these limitations do not significantly affect the quality of our manually segmented corpus since only the predictive power of the language model for the complete input strings is important instead of the particular segmentation.

Based on the list of segmented words, we automatically substituted each word in the corpus (from which the word type list for the manual segmentation was derived) with its segmentation, thus producing a linguistically morpheme-segmented corpus (manually_seg). It consists of 1,141,434 tokens and 11,154 distinct morphemes.

5.2 Comparison of linguistic morpheme-based language model with word- and statistical morpheme-based language models

To find out whether the manual segmentation contributes to an improvement of model quality, we also segmented the same text corpus using Morfessor, and consequently obtained a statistically morpheme-segmented version of the corpus (morfessor_seg). The same text corpus (without being morphologically segmented) has also been used to train word-based models. That means we have three versions of the corpus: an unsegmented version for training word-based models and a linguistically morpheme-segmented as well as a statistically morpheme-segmented ones. Each of the three corpora has been divided into three parts: training, development, and test sets in the proportions of 80:10:10. Language models with several smoothing techniques and n-gram orders have been trained on them. The best model for each corpus is presented in Table 10.

Table 10.

Perplexity difference according to the type of corpus

The models developed with manually segmented data outperform the ones developed with automatically segmented data in terms of perplexity.Footnote ¹² If, however, the log probability is considered, the word-based model still seems to fare better than the morpheme-based ones. With regard to OOV words, the models developed using automatically and manually segmented corpora led to lower OOV rate. In contrast, the number of OOV words in word-based models is very high. One can also see that Unmodified Kneser–Ney smoothing technique performed well irrespective of the corpus/data.

Interpolation improved (see Table 11) all the models regardless of the kind of corpus used. However for the word-based model, the increase in log probability is not as large as it is for the others. The language models developed using manually segmented data consistently outperformed (in terms of perplexity) the ones developed using automatically segmented data.

Table 11.

Effect of interpolation

6.1 The speech corpus

The speech corpus used to develop the speech recognition system is a read speech corpus (Abate et al. Reference Abate, Menzel and Tafila2005). It contains 20 hours of training speech collected from 100 speakers who read a total of 10,850 sentences (28,666 tokens). Compared to other speech corpora that contain hundreds of hours of speech data for training, this corpus is obviously small in size and accordingly the models will suffer from a lack of training data. Again, the 5k development set has been used for the purpose of this investigation.

6.2 Word-based recognizers

6.2.1 Acoustic, lexical, and language models

The acoustic model consists of 6,610 crossword triphone Hidden Markov Models (HMMs) each with three emitting states. The states of these models and all the crossword triphone models that are potentially needed for recognition are tied using decision-tree-based state-clustering that reduced the number of triphone models from 77,658 to 10,215. Their mixture is added incrementally and 12 Gaussian mixtures have been found to be optimal.

The vocabulary for the three full-word form pronunciation dictionaries has been prepared by taking the most frequent words from the ATC_120k text corpus described in Section 3.1. Table 12 shows the OOV rates on the 5k development set that consists of 360 sentences (4,106 tokens and 2,836 distinct words) for these vocabularies. Although we tried to optimize the vocabularies by taking the most frequent words, the OOV rate is still high. The pronunciation dictionaries have been encoded using a simple procedure that takes advantage of the orthographic representation which is a consonant vowel syllable.

Table 12.

Word OOV rate on the 5k development set

The text corpus from which the vocabularies have been selected has also been used to train the language models. As there are three dictionaries (5k, 20k, and 65k), we have developed three trigram language models one for each vocabulary using the SRILM toolkit (Stolcke Reference Stolcke2002). The language models are made open by including a special unknown word token. The Modified Kneser–Ney smoothing method has been used to smooth all the language models.

6.2.2 Performance of word-based speech recognizers

Speech recognition experiments have been performed using the 5k, 20k, and 65k vocabularies. In each case the system has been evaluated with the 5k development set. The results are reported in terms of WER and syllable error rate (SER). SER has been used to provide a fair comparison with the morpheme-based recognition performance as the WER of these systems might be affected by the generation of illegal morpheme sequences. Figure 4 presents both WER and SER for each vocabulary. As it can be noted, both, the OOV rate and the error rates decrease as the vocabulary size grows. The best performance (21.5% WER or 8.1% SER) has been obtained for the 65k vocabulary which has the smallest OOV rate. The results show that the OOV rate highly affects the performance of a speech recognition system.

Fig. 4.

Recognition error rate of word-based recognizers.

6.3 Morpheme-based recognizers

To deal with the problems faced with a word-based speech recognition system for Amharic, such as the effect of OOV rate (see Section 6.2.2), morphemes instead of words have been considered as dictionary entries and modeling units for the language models.

6.3.1 Acoustic, lexical, and language models

The acoustic model has been developed in a similar fashion as for the word-based recognizers. The training data is covered by a set of 6,459 cross-morpheme triphone HMMs each with three emitting states. The states of these models and all the possible cross-morpheme triphone models are tied and, therefore, the number of triphone models is reduced from 57,799 to 7,685. Similar to the word-based models, 12 Gaussian mixtures have been found to be optimal.

The Morfessor segmented ATC_120k corpus (see Section 3.3) has been used in the development of acoustic, lexical, and language models of the morpheme-based recognizers. In order to facilitate the conversion of morpheme sequences to words, a special word boundary marker has been attached to word boundary morphemes which made the morphemes context-sensitive and consequently increased the number of distinct morphemes to 28,492. From the morphologically segmented corpus, three dictionaries have been prepared: 5k and 20k by selecting the most frequent morphemes and 28.4k by considering all of them. The morpheme-based OOV rates of these vocabularies on the 5k development set are presented in Table 13 which shows that the OOV rate is considerably reduced as a result of using morphemes.

Table 13.

Morpheme OOV rate on the 5k development set

In order to analyze the benefit of morpheme-based system in terms of covering OOV words, we calculated a word-based OOV rate by counting the number of words which can not be rewritten in terms of the morphemes found in the vocabularies. Table 14 shows the word-based OOV rate of the morpheme-based systems. The result is comparable to the content of Table 12. As can be seen, the use of morphemes reduced the OOV rate for words to a similar degree. The token OOV rate for the 5k morpheme vocabulary, for instance, is reduced by almost 50% from the token OOV rate of the 5k full-word vocabulary (cf. Table 13).

Table 14.

Word OOV rate of morpheme-vocabularies on the 5k development set

Using the three dictionaries (5k, 20k, and 28.4k), we have developed three open vocabulary morpheme-based trigram language models, one for each vocabulary. Similar to the word-based language models, the morpheme-based ones have also been smoothed using Modified Kneser–Ney smoothing technique.

6.3.2 Performance of morpheme-based speech secognizers

The morpheme-based speech recognition system has been evaluated on the 5k development set using the 5k, 20k, and 28.4k morpheme vocabularies. The results are reported in terms of morpheme error rate (MER), WER, and SER. The WER has been computed after words have been obtained by concatenating the recognized morpheme sequence. The best performance (see Table 15) has been obtained with the 28.4k morpheme vocabulary which has an OOV rate of 0.03. Since the OOV rate is very small, an even better performance than the one reported here was expected. The reasons for this disappointing result (in spite of having a small OOV rate) might be a higher acoustic confusability and the limited language model scope which are peculiar to morpheme-based speech recognition systems.

Table 15.

Performance of morpheme-based speech recognizers

6.4 Comparison of word- and morpheme-based speech recognizers

The very low OOV rate of morpheme vocabularies has a positive effect on speech recognition accuracy, especially for small vocabularies, namely the 5k one. As can be seen in Tables 15 and 16,Footnote ¹⁵ while the word-based model produces a WER of 61% and SER of 25.8% for the 5k vocabulary, the morpheme-based system improves WER and SER to 49.2% and 19.7%, respectively, for the 5k morpheme vocabulary.Footnote ¹⁶ This corresponds to an absolute WER reduction of 11.8% and an SER reduction of 6.1%. The error rate reduction is statistically significant (for both WER and SER) with p<0.001.

Table 16.

Performance of word-based speech recognizers

The 20k morpheme-based speech recognizer performed slightly worse (with 37.5% word and 15.1% SER) than the equivalent word-based system which has a WER of 37.1% and a SER 14.6%. However, there is no statistically significant difference between the two systems at p=0.05. The performance of the recognizer with the 28.4k morpheme vocabulary is only slightly better than that of the 20k word-based recognizer although it includes all the morphemes in the text and has a very low OOV rate. As we have already mentioned, besides the acoustic confusability, the limited scope of the morpheme-based n-gram language model might contribute to the poor performance of the morpheme-based speech recognizer.

To see the effect of acoustic confusability, we have computed, using National Institute of Standards and Technology (NIST's) scoring package (sclite),Footnote ¹⁷ the statistics of the confusion pairs for the best performing word- and morpheme-based recognition systems. The result shows that there are a total of 677 confusable word pairs out of which only 2 have a frequency of 3 and only 7 have a frequency of 2. On the other hand, there are a total of 1,376 confusable morpheme pairs and the frequency of confusion goes up to 13. There are 4 and 11 confusable morpheme pairs with frequency of 4 and 3, respectively. We have also seen 32 confusable morpheme pairs with frequency of 2. The summary is given in Table 17.

Table 17.

Acoustic confusion of word- and morpheme-based ASR

The problem of acoustic confusability has also been commented on by Pellegrini and Lamel (Pellegrini and Lamel Reference Pellegrini and Lamel2006) who suggested the use of higher order n-gram models. Accordingly, we also experimented with higher order morpheme-based language models in our morpheme-based speech recognizers.

Since it is not possible to use n-gram models higher than trigram in the HTK decoding tool, we generated lattices using the 20k and 28.4k vocabulary morpheme-based recognizers and rescored the lattices with a quadrogram morpheme-based language model which has been developed in the same manner as the trigram models. As can be seen from Table 18, a 1.2% absolute WER reduction (compared to the 20k word-based recognizer) has been obtained for the 20k vocabulary morpheme-based recognizers. However, this improvement is statistically insignificant at p=0.05. With respect to SER, the word-based system still performs slightly better than the morpheme-based one (14.6% versus 14.7%). The accuracy of the 28.4k system which has a very low OOV rate (0.03) and covers all the morphemes in the text is still worse than the accuracy of the 65k word-based recognizer (21.5% WER or 8.1% SER) which has a considerably higher OOV rate (9.33). On the other hand, the result is still better than the one achieved with a trigram morpheme-based language model. This might indicate that acoustic confusability is the most influential factor for the performance degradation of a morpheme-based speech recognizer on large vocabularies.

Table 18.

Lattice rescoring with a quadrogram morpheme-based language model

We conclude from these experiments that the use of morphemes as lexical and language modeling units instead of words contributes to a reduction in the number of tied acoustic models, a lower OOV rate of the lexical model and a lower perplexity of the language model. The cumulative effect results in a better performance of the morpheme-based ASR and a higher decoding speed.

As it has been given in Sections 6.2.1 and 6.3.1 the number of cross-word triphones is 10,215, while that of cross-morpheme triphones is 7,685. To see the effect of the difference in the acoustic models on the ASR performance, we have compared the performance of 5k word- and morpheme-language models on both acoustic models. The result shows that with 10,215 cross-word triphone models, the WER of the morpheme-based language model is 42.6% while that of the word-based language model is 61.0% making their absolute difference 18.4%. With 7,685 cross-morpheme triphone models, the WER of the morpheme-based language model is 49.2% while that of the word-based language model is 66.4% that makes their absolute difference 17.2%. This shows that in any of the acoustic models the morpheme-based language model scores a better performance than the word-based language model.