2.1 Strong verbs

Strong verbs are those without any weak radical. They can be categorized into three classes: regular, hamzated, and doubled. Regular and doubled verbs can be generated using well-behaved morphological regular rules, while hamzated verbs are irregular ones. The hamza letter (‘’) is changed to other different realizations due to the influence of the vowels before and after the hamza (Al-Jumaily et al. Reference Al-Jumaily, Martínez, Martínez-Fernández and Van der Goot2011). The different realizations of the hamza letter are () / >, <, &, ‘,} /. For example, the conversion of () / > / into () /&/ can be explained by taking the imperfect tense with passive voice for the root / > -k-l/. Applying regular rules would generate /yu- > okal/ but as a hamzated verb it should be generated as /yu-&okal/ (be eaten).

2.2 Weak verbs

Weak verbs, a major concern of this work, can be categorized into three classes: assimilated, hollow, and defective.

Assimilated verbs are the verbs with a weak initial radical. They fall into two categories: verbs beginning with letter () /y/ such as “” /ya}isa/ (despaired), and verbs beginning with letter () /w/ such as “” /waEada/ (promised). The perfect and imperfect stems of the first category are generated by the same rules of strong verbs, while the imperfect stem of the second category is generated by the following specific rule:

• • The initial radical () /w/ is deleted in the imperfect form if its middle short vowel is () /i/. For example, the imperfect conjugation of the root /w-r-v/ is /ya-riv/ (he/it inherits), while for the root /w-j-l/ it is /ya-wjal/ (he/it is afraid).

  Hollow verbs are the verbs with a weak middle radical. They fall into the following four categories (Cavalli-Sforza, Soudi and Mitamura Reference Cavalli-Sforza, Soudi and Mitamura2000; Soudi, Cavalli-Sforza and Jamari Reference Soudi, Cavalli-Sforza and Jamari2001):

• • Verbs where the middle radical is () /w/ of the pattern /faEala/ such as /Sawama/ ‘fasted’ or /faEula/ such as /Tawula/ ‘he/it was long’.

• • Verbs where the middle radical is () /w/ of the pattern /faEila/ such as /xawifa/ ‘was afraid or frightened’.

• • Verbs where the middle radical is () /y/ of the pattern /faEala/ such as /sayara/ ‘walked’.

• • Verbs where the middle radical is () /y/ of the pattern /faEila/ such as /Hayira/ ‘was confused or hesitated’.

In both perfect and imperfect tenses, the stem is realized by either a short or a long vowel depending on person, number, and gender features (Cavalli et al. Reference Cavalli-Sforza, Soudi and Mitamura2000). For example, to generate the third person masculine plural imperfect stem of the root /S-w-m/, a long vowel is used such as /ya-Suwmu-wna/ (they fast), while to generate the third person feminine plural imperfect stem of the same root, a short vowel is used /ya-Sumo-na/ (they-(f) fast).

Defective verbs are the verbs with a weak final radical. They fall into the following five categories:

• • Verbs where the final radical is () /w/ of the pattern /faEala/ such as /najawa/ ‘was rescued’.

• • Verbs where the final radical is () /w/ of the pattern /faEila/ such as /raDiwa/ ‘was pleased’.

• • Verbs where the final radical is () /w/ of the pattern /faEula/ such as /saruwa/ ‘was noble’.

• • Verbs where the final radical is () /y/ of the pattern /faEala/ such as /ramaya/ ‘threw’.

• • Verbs where the final radical is () /y/ of the pattern /faEila/ such as /xaziya/ ‘was despicable or ashamed’.

In both perfect and imperfect tenses, the stem is realized by either a short or a long vowel depending on the person, number, and gender features. For example, to form the third person masculine plural perfect stem of the root /n-j-w/, a short vowel is used such as /naja-woA/ (they were rescued), while to form the first person singular perfect stem of the same root, a long vowel is used /najawo-tu/ (I was rescued). The imperative tense can be generated from imperfect tense by adding () /{/ at the beginning of a verb.

3.1 Lexical error analysis and correction within spelling checkers

Spell checkers are very successful at detecting and correcting basic error operations. In a written language, any misspelling can be described as a transformation from a correctly spelled word performed by one or several of the basic error operations (Ingels Reference Ingels1997):

• • Deletion (e.g. * instead of /{arju / I-hope)

• • Insertion (e.g. * instead of /{ihtam ~ a/ interested)

• • Substitution (e.g. * instead of /bada > a/ started)

• • Transposition (e.g. * instead of /naqala/ or /nuqila/ transfer or was transferred)

The basic error operations can be used to describe lexical errors, but they are not suitable for explaining them (Shaalan, Allam and Gomah Reference Shaalan, Allam and Gomah2003). Lexical errors can be classified according to the linguistic knowledge of a typist into the following types:

• • Typographic errors, ‘performance errors’: The typist knows the correct spelling of the word but still makes a simple error. For example, the substitution of ‘’* (the-man-stood-up) for ‘’ /qAma Alrajul/ (the-man stood-up) is a typical typographic error caused by skipping the space between two words.

• • Cognitive and phonetic errors, ‘competence errors’: The typist lacks some linguistic competence (knowledge) of the language. For example, the substitution of closely pronounced letters as in ‘’* / > u-tiyE/ instead of ‘’ / > u-TiyE/ (I obey) is a typical cognitive error caused by substituting the letter ‘’ /t/ with the letter ‘’ /T/.

Another important classification of lexical errors is whether the misspelled word is either (Shaalan and Attia Reference Shaalan and Attia2012)

• • unknown ‘isolated error’. For example, the substitution of the misspelled word ‘’ /qulo-tw/ with ‘’ /qulo-tu/ (I said) is a typical isolated error caused by inserting a letter instead of a diacritic sign that gives the same pronunciation, or

• • known but refers to unintended word ‘context sensitive error’. For example, the substitution of the word ‘’ / > a-tawaq ~ aE/ (I expect) with the word ‘’ / > u-waq ~ iE/ (I sign) is a context sensitive error where the insertion of a letter, i.e. /t/ as a second letter, changes the word meaning.

Either rule-based shallow analysis or frequencies of bigrams and trigrams can be used to detect isolated errors (Attia et al. Reference Attia, Pecina, Samih, Shaalan and Genabith2012). Words that contained character bigrams or trigrams of low frequency were judged to be (possibly) incorrect (Daciuk Reference Daciuk1998). This approach requires calculations of word frequencies from an available corpus. Full parsing or detailed analysis would be an ideal solution to detect context sensitive errors (Bigert Reference Bigert2004). The words that do not fit into the grammar are considered misplaced.

In agglutinative languages, such as Turkish, and semitic languages, including Arabic, it is impossible to base a spell checker on a word list as is often done with morphologically poor languages such as English and French, given the highly inflectional and derivational morphological nature of these languages (Shaalan et al. Reference Shaalan, Allam and Gomah2003). To detect misspelling errors; the program would simply try to morphologically analyze the input word; if it fails, the word is probably misspelled (Shaalan, Aref and Fahmy Reference Shaalan, Aref and Fahmy2010c).

Traditional spell checkers, such as those found in present day word processors, include a spelling correction component that suggests correction as a feedback to the end user. The list of suggestions includes words present in the dictionary that are similar to the erroneous word in a certain aspect. This similarity can be measured in a number of ways, including edit distance or similarity of roots in terms of morphological categories or pronunciation (Daciuk Reference Daciuk1998). The edit distance technique is used to measure the similarity of two strings by counting the minimal number of character insertions, deletions, or substitutions needed to transform one string into another. It has been used to only correct typographic errors.

3.2 Lexical error analysis and correction within intelligent language tutoring systems

This research addresses the development of an ILTS. One aim of this kind of systems is to enhance the teaching and learning of foreign languages (Rimrott Reference Rimrott2005). This software can support listening, speaking, reading, and writing skills in a foreign language. Typically, ILTS provides the learner with individualized teaching and flexible feedback. It usually contains exercises for language learners whose responses are analyzed by the system in order to provide some form of detailed feedback.

Most Arabic intelligent language tutoring systems developed until now try to overcome shortcomings of general spell checkers (Shaalan et al. Reference Shaalan, Magdy and Samy2010c). They do so by incorporating morphological knowledge and non-native intuitions into their algorithms in order to be able to handle competence errors made by non-native writers. However, true diagnosis indicating the precise reason of the error is very rarely done. They still keep the behavior of spell checkers by only offering a short list of alternative words to replace the unknown erroneous words.

Shaalan (Reference Shaalan2005a; Reference Shaalan2005b) developed an ICALLFootnote ⁵ system for Arabic learners which employs deep processing and implementation of mal-rules (buggy rules) to provide error-specific feedback to morpho-syntactic errors. These specific rules make it possible to parse erroneous expressions or structures for the sake of error diagnosis. This proposed work still represents a state-of-the-art of grammatical checking within Arabic ICALL systems. The objective test method was adopted to assess learners’ knowledge. There were two main types of test items for learner's interaction: selection type, requiring the learner to select the correct answer, and supply type, requiring the learner to write few words. In comparison with our paper, this research is simple and restricts exercises to those for which we have one and only one correct answer. While this work is admirable in many respects, error modeling and diagnosis using deep processing is limited to morpho-syntactic errors. Also, the feature set is considered simpler than the ones we used in error analysis. In our paper we investigated advanced issues that show the benefits of using deep morpho-syntactic processing.

Habash and Roth (Reference Habash and Roth2011) employed deep morphology to develop a general-purpose Arabic handwriting error recognition tool. A learning algorithm is used to generate a predictive model from linguistic and non-linguistic features. The handwritten text is input to an Optical Character Recognition (OCR) device and the handwritten recognition software is applied to the scanned image to convert it into text. Then the model is applied to each word in the input text in order to predict illegal words. This tool might be useful for Arabic intelligent language tutoring systems in order to guide the identification of errors in handwritten learner's answer. In comparison to Habash and Roth (Reference Habash and Roth2011), our paper is about deep rule-based error analysis that uses rich linguistic features and provides adaptive feedback messages according to the learner's level.

Alfaifi and Atwell (Reference Alfaifi and Atwell2012) have conducted a study about Arabic learner corpora. They proposed design criteria for Arabic learner corpora that include determining the participants, size of the corpora, data collection approach, selection method of representative textual data (teaching and learning material), and taxonomy of learner's errors and their tag set. The study analysis is applied on Arabic learners from Saudi Arabia and reveals that the suggested error taxonomy is more appropriate for errors’ markup in Arabic corpora due to its accuracy and comprehensiveness. The Arabic learner corpora are not yet ready. The availability of such linguistic resource would impact not only the development of Arabic error analysis and diagnosis systems but also the performance evaluation with existing systems because we will be able to do a fair comparison based on standard training/test sets and well-defined target (i.e. errors and their corrections).

To this end, Arabic intelligent language tutoring systems are still in their initial stages compared with the work done on other languages, such as English, from which we can benefit due to the extensive research done in this field. Hsieh et al. (Reference Hsieh, Tsai, Wible and Hsu2002) developed an ILTS for English lexical errors. The system provides an open writing environment. They allow learners to write freely without an exercise model at the beginning. They categorized English lexical errors into three classes: compound, morphological, and spelling errors. To identify compound errors, they pre-process each compound word in the dictionary and store it in a compound word table (CWT). If the input word is in the CWT, their system recognizes it as a compound error. Morphological errors can be recognized and corrected using the reduction-inflection (R-I) algorithm. The basic idea is to reduce the input word to its base form according to several reduction rules, inflect the base form to its inflected forms, and compare the inflected forms with the input. Finally, they used minimum distance techniques to identify and correct spelling errors.

Rimrott (Reference Rimrott2003) developed a prototype program called ‘SANTY’ that provides more adequate spelling suggestions to non-native speakers of German by incorporating morphological knowledge as well as knowledge of common types of learner's mistakes into its spell-checking algorithm. The program treats predictable verb inflection mistakes. It uses morphological analysis to determine the intended verb inflection based on the wrongly inflected input word.

The research done by Faltin (Reference Faltin2003) and Faltin, L'haire and Ndiaye (Reference Faltin, L’haire and Ndiaye2005) has presented an ILTS for language learners of French. Three different methods were used to retrieve correction proposals: alpha-codes, phonological reinterpretation, and ad hoc rules (pending the implementation of a morphological analyzer). Alpha-code is a simplified representation of a word, obtained by reordering the characters it is composed of, stripping the word of its diacritics, and getting rid of duplicated characters. Words sharing a single alpha-code are similar in some ways. Spelling errors that can be treated by this method are inverted letters, incorrect diacritics (including multiple incorrect diacritics) as well as some missing or superfluous characters. Phonological reinterpretation bases the similarity criteria on the sounds of the language. An expert system is used to transform the written word into its phonological representation. It is then easy to search a lexicon indexed by pronunciation for all the words that sound alike. Ad hoc rules allow their system to find adequate correction proposals for a morphological error. This is a very limited technique, spot illegal words ending in -als or -ails. The system replaces the incorrect ending by the correct one and checks that the newly composed word is a true word before displaying it as a correction proposal. The system finally ordered the results proposed by three methods. It presents results of ad hoc rules, followed by phonological reinterpretation, showing last the alpha-code.

6.1 Item banking

The item banking is a database of test items such that each question in this database is linked to some learning section in the domain model. This component is used to generate different types of test items each time the learner is asking to take a test. The material generator component receives learner request and then selects test items in a random order.

There are two main types of test items for interaction with learners in Arabic ILTS: supply-type and selection-type. For supply-type items, the learner answers one or more words, such as a word in ‘fill-in-the-blank’ or a short sentence from a given sequence of roots. For selection-type items, the learner selects the answer from a pool of answers such as ‘which word is different?’ or ‘multiple-choice’. From the linguistic point of view, the type of exercises covered by Arabic ILTS can be classified as follows:

1. (1) Supply-type questions, including the following:

   • • Dictation. This exercise type focuses on listening skills of the learner. Learners should first listen to an Arabic sentence, then transcribe it. Next, the system diagnoses the learner answer. If it is correct, a positive feedback message is returned. Otherwise, an error-specific feedback message is provided. The learner has two additional options: either correct the error and resubmit the sentence again or peek at the correct answer.

   • • Word order. This exercise displays a number of Arabic roots, the learner task is to rearrange these roots and conjugate them to form a grammatical Arabic sentence.

   • • Write a sentence. In this exercise type, learners are provided with a sequence of Arabic roots. Their task is to generate a grammatical Arabic sentence using all the provided roots in the given order. The learner is warned for any extra or missing words.

   • • Transform the sentence category. Arabic has two broad categories of sentences: nominal and verbal sentences. So, in this exercise type, learners are provided with some sentences in one category and their task is to transform these sentences into the other category.

   • • Word formation practice. This exercise displays some Arabic roots and the learner task is to conjugate these roots according to a given form.

   • • Fill in blank. The learner's task is to complete sentences by filling in any blank with a correct verb conjugation. The learner has to use a given set of roots.

2. (2) Selection-type questions, including the following:

   • • Which word is different? This exercise displays a list of words such that all except one share the same features. The learner's task is to identify the divergent that differs morphologically from other convergent words in some aspects, such as root, pattern, verb tense, voice, and type of weak verb.

   • • Multiple choice. The learner's task here is to choose the correct word that matches the given features from multiple choices. The choices are different conjugations of the same word.

The structure of the item banking consists of four database relations: question, question content, answer, and concept relations.

Each question is accompanied by an associated list of conceptsFootnote ⁸ to test and keep a record of how well the learner has mastered them. Furthermore, each question has specific parameters that help the system in diagnosing errors. These parameters depend on the type of the test item. The set of parameters of the supply test item type is a list of feature structures (FS); each of which describes an Arabic word in the correct answer. Each FSFootnote ⁹ includes the following features: correct word without diacritics, correct word with diacritics, root, pattern, type of verb, prefix string, suffix string, lexical category, tense, voice, mood, type,Footnote ¹⁰subject, object gender, number, and person.

For the selective test item, the set of parameters depends on the exercise type. In ‘multiple choice’ questions, the associated set of parameters is a list of feature structures that describes the intended features of each word in the answer. Each FS includes the following features: word, root, lexical category, pattern, tense, voice, mood, subject person, gender, and number. In ‘which word is different?’ question, the associated set of parameters is the intended aspect of divergence (e.g. root, pattern, lexical category, tense, voice, type of weak verb) and the value of the divergent feature for each word in the question.

6.2 Student model

The student model used in Arabic ILTS keeps a record of the current learning skill levels of the student (domain-specific knowledge). The student learning skill levels are represented by the frequency distribution of committed errors with regard to their concepts. Therefore, we calculate the individual learning skill level in terms of the frequency of an error that the student has made while fulfilling its related concept. For example, the learner may be a beginner with regard to ‘incorrect use of root pattern’ error but advanced with ‘make a long vowel short’ error. This knowledge is represented by a perturbation error model. In this model, we assume the presence of one or more misconceptions for each concept that are related to teaching of Arabic weak verbs. By this way, the student model knowledge is represented by the union of a subset of the domain knowledge (Arabic concepts of weak verbs) and another subset of the misconception set. In the proposed Arabic ILTS, the student model is a database of error records. Table 5 shows an extraction of the student model database table which defines that the vowel letters concept (ID = 2) has two associated misconceptions: make a short vowel long and vice versa. The complete student model database lists all Arabic concepts and their associated misconceptions defined in the domain model.

Table 5.

An extraction of the student model database table

Another databaseFootnote ¹¹ table showing student skill is used to express current skill of each student.Footnote ¹² The frequency of the error is expressed by a number pair, i.e. (EFC, TC), where EFC is the Error Frequency Counter variable that indicates how many times the student has made this error, and the TC variable is the total counter that indicates the total number of times for which the student has met this concept. As the frequency of the error becomes higher, the student requires detailed instruction. For example, Table 6 shows that the student (ID = 2012123) has made the error that replaces the short vowel with the long one three times while s/he was practicing the vowel letters concept four times. Therefore, for each error of certain concept, the student model keeps a proportion of its frequency to the total times the concept has been met such that the ratio falls in the range of one of the three learning skill levels. The learning skill level determines the level of details of the feedback. We decided to determine the student learning skill level as follows:

Table 6.

An extraction of the student skill database table

• • Beginner: Frequency of the error > 0.50.

• • Intermediate: 0.20 < frequency of the error ⩽ 0.50.

• • Advanced: 0 ⩽ frequency of the error ⩽ 0.20.

Hence, in Table 6 the learning skill level of the student 2012123 with regard to make a short vowel long error is 75 percent because he did this error three times out of four. This is interpreted as, the student's skill is still at the beginner's learning level.

6.3 Expert model

This component is responsible for analyzing learner's answer, detecting possible error types, and accumulating the frequency counters. The expert model performs NLP tasks through Arabic computational tools: morphological analyzer, morphological generator, and error analyzer. In particular, the expert model retrieves from the Item Bank the intended features of each word in the question (i.e. question's parameters), concepts associated with each question, and the test item type. Then, according to the test item type, it proceeds with the appropriate procedure for analyzing the learner's answer.

As the correct answer of the selective test item type is predetermined, the processing does not need deep Arabic NLP. However, it is needed for the supply test item type in order to get all possible analyses of the learner's answer and then to select the most appropriate one according to the instruction learner level and difficulty of Arabic conceptsFootnote ¹³ (cf. Shaalan et al. Reference Shaalan, Magdy and Fahmy2010a; Reference Shaalan, Magdy and Samy2010b). For example, the passive Arabic verb might have the same orthographic form as the active one. However, it is doubtful that a beginner learner of Arabic would use a passive voice verb instead of an active voice. So it is obvious in this case that passive voice is a rare construction. Hence, we allow some prioritized conditions in order to select the most preferred word analysis, such as ‘if the question goal is to test passive voice then the system selects passive voice analysis otherwise, it selects the active voice analysis’.

6.4 Tutoring module

This module is responsible for initializing the student model. When a new learner is registered, all the frequency counter pairs are set to [0, 0], which assigns advanced level to the learner. Further practice might step him/her down to either beginner or intermediate learning level.

The feedback system is also responsible for issuing appropriate error-specific feedback message suited to the learner's expertise. There exist three levels of errors that correspond to the following learning skill levels kept in error database relations: error class, error type, and source of errors. An example of the error class entry is the ‘word formation due to phonology’ (Arabic translation ) lexical error. An example of the error type entry related to this class is ‘vowel letter’ (Arabic translation ).

Examples of the source of errors that belong to the same error type are: ‘make a short vowel long’ (Arabic translation: ) and ‘make a long vowel short’ (Arabic translation: ), which belong to the ‘vowel letter’ error type.

The feedback message generator module receives a number between 1 and 3, indicating the learning skill level according to this error and the error type along with its parameters.

The general procedure for feedback generation works as follows: For the advanced learning level, the feedback is most general, providing a hint to the class of error. For the intermediate learning level, the feedback is more detailed, providing additional information on the type of error. For the beginner, the feedback is the most precise one. It not only pinpoints the class and type of the error but also refers to the source of the error. For example, if the learner writes the word * /zwro-tu/ (I-visited) instead of /zuro-tu/ (I-visited), the source of the error here is ‘make a short vowel long’. The expert model detects this error and sends to the ‘vowel letter’ error type of tutoring module with the following parameters: [‘short’, ‘long’]. The feedback generation module checks the learner skill level. For advanced learner, it issues the following error class message: ‘’ (word formation due to phonology). For intermediate learner, it issues the following error class message: ‘’ (vowel letter). For beginner learner, it issues the following source of error: ‘’ (make a short vowel long).

It is worth noting that there is a specific procedure for the feedback message generation of ‘which word is different?’ test item question. It works as follows: For the advanced learning level, the feedback is most general, providing a hint to the type of the error (i.e. the divergent aspect such as root, pattern, lexical category, tense, voice, and the type of weak verb). For the intermediate learner, the feedback is more detailed, providing additional information on the value of divergent feature for learner answer. For the beginner learning level, the feedback is the most precise one. It not only pinpoints the type of the error but also refers to the exact source of the error (the value of the divergent feature for both correct and learner answer). For example, consider that the divergence aspect is ‘type of weak verb’ () when all choices in the question are defective perfect verbs except one, which is a hollow perfect verb. Then, by selecting a defective verb, the source of error becomes ‘’ (confusion between defective and hollow verbs). The expert model should detect this error and send the tutoring module the error type ‘type of weak verb’ with the parameters: defective and hollow. The feedback generation module checks the learner skill level. For advanced learner, it looks up the error type database and issues the corresponding error message: ‘’ (wrong selection: the error is in the type of a weak verb). In the case of intermediate learner, it issues a combination of the error type and the value of the divergent feature error class message: ‘’ (wrong selection: the error is in the type of a weak verb. This word is a defective verb). Finally, for beginner learner it issues the following source of error: ‘’ (Wrong selection: The error is in the type of the weak verb. This word is a defective verb, while the correct answer is a hollow verb).

7.1 Error diagnosis of selective test item

In the selective test type, error diagnosis is straightforward. It is guided by the feature structures describing the main properties of each word in the answer. For example, the following is the FS of the well-formed word

/ > a-quwl/ (I-say):

The comparison of feature-value pairs between the correct FS and the erroneous FS leads to identifying the error type.Footnote ¹⁷ The tutoring module receives as input the error type, error parameters, and learning skill level of this error, and produces as output the appropriate feedback. For example, consider the following ‘multiple choice’ question, which asks the student to select a correct answer from a given list of words.

Example 5: Choose the correct answer from the words between parentheses:

The correct answer is / > a-quwl/ (I-tell). If the learner answers /qAla/ (told), a perfect verb derived from the root /q-w-l/ and pattern /faEala/, then this answer will be diagnosed as a verb tense error. The parameters that indicate the mismatches are: imperfect active verb in indicative mood form in the correct answer and perfect active verb in the learner's answer. The tutoring module issues the feedback message that is suitable for the error learning skill level:

• • Advanced learner: ‘’ (morphological or syntactic context sensitive error),

• • Intermediate learner: ‘’ (verb tense error), or

• • Beginner learner: ‘’ (incorrect use of perfect active verb instead of imperfect active verb in the indicative form).

In the ‘which word is different?’ question, the parameters are the aspect of divergence (such as root, pattern, lexical category, tense, voice, type of weak verb), and the possible values. For example, consider the following question, which asks the student to select the word that differs from the other words.

Choose the divergent word:

• • Summon

• • Strived

• • Threw

• • told

The correct answer is the divergent word /qAla/ (told). All convergent words are defective (a weak final radical) perfect verbs, whereas the divergent word is a hollow (a weak middle radical) perfect verb. In this case, the aspect of divergence, the type of weak verb, is the error type. The parameters that indicate the mismatches are: hollow verb in the correct answer and defective verb in the learner's answer. The tutoring module issues the feedback message that is suitable for the error learning skill level:

• • Advanced learner: ‘’ (wrong selection: the error is in the type of a weak verb).

• • Intermediate learner: ‘’ (wrong selection: the error is in the type of a weak verb. This word is a defective verb), or ‘’

• • Beginner learner: ‘’ (Wrong selection: The error is in the type of a weak verb. This word is a defective verb, while the correct answer is a hollow verb).

7.2 Error diagnosis of a supply test item

Error diagnosis of a supply test item – where the learner is asked to write a few words – is more sophisticated than a selective type, since the student is allowed to be more creative. So it is necessary in the proposed Arabic ILTS to incorporate morphological knowledge and non-native intuitions into its error diagnosis algorithm in order to be able to handle competence errors made by non-native learners. The preconditions for error analysis are concerned with verifying whether the learner has overlooked, inserted, or modified the required order of any word in the input. The learner should correct the input and make sure that the answer is free of such warnings. The current system is organized in such a way that only one warning at a time is reported to the learner. Example 6 shows answers that cover these errors. The rest of this section gives more insights on the diagnosis steps followed by the system.

Example 6: Write a sentence using the following roots and follow the given right-to-left order.

• /d-w-m / /H-q/ /q-w-l/.

There are three roots. Assume the following five possible answers.

1. a) / > a-quwl AlHaq ~ dA}imAF/ (I always tell the-truth).

2. b) / > a-quwl dA}imAF/ (I always told).

3. c) / > a-quwl AlHaq ~ dA}imAF muxoliSAF/ (I always tell the-truth faithfully*).

4. d) /dA}imAF qAlo-tw AlHaq ~ / (always, I-told* the-truth)

5. e) * /qAlo-tw AlHaq ~ dA}imAF/ (I always told* the-truth).

Answer (a) is a correct answer. In answer (b), the second root ( /H-q/) is missing. In answer (c), the learner has added an extra word ( /muxoliSAF/) at the end of the sentence. In answer (d), the learner has not followed the required word order. In answer (e), the learner input has met the conditions of not missing a given word, adding extra word(s), and not violating the word order. However, s/he has made lexical errors in the first word /qAltw/ (I-told). The lexical error analysis performs a comparison between features of words in the correct answer and features of words in the input, and concludes that in answer (e) the learner has made three lexical errorsFootnote ¹⁸ with the word /qAlo-tw/ (I-told). The following section uses this answer, as a working example, to show how the word and lexical error analyzers are able to diagnose these errors. It is difficult to have one single example that demonstrates the functionality of each diagnosis step and how it causes the solution structure to evolve. So, for completeness, we tried our best to explain each step in the error diagnosis algorithm using answers in Example 6. Besides, we give additional examples and explanations as necessary.

7.2.1 Identification of base forms

The result of the match between the features in the learner's answer with the retrieved features of the correct answer is exact matches and mismatches. We are interested in analyzing the latter, so we delete the exact matches from the input. Then the word analyzer applies constraint relaxation and edit distance techniques on each word in the learner's answer and produces all possible base word forms. The base word form is defined as a normalized root form, i.e. the root after removing any weak or hamza letters, which facilitates the comparison among different verb conjugations that are derived from the same root without doing extra processing for recognizing the lexographic change in such irregular forms. For example, the base form of both / > a-quwl/ (I-tell) and /qAla/ (told) is /q-l/. The base form is used to verify whether the student overlooked a given word or introduced a new word in his/her answer.

The word analyzer also suggests initial error detection hypotheses that will be confirmed later on by the lexical error analyzer. The constraint relaxation and edit distance techniques split the erroneous word into three segments: prefix + stem + suffix. With the constraint relaxation technique the partial structures can combine only if certain constraints imposed by the language are met. When these constraints are relaxed, an attachment is allowed even if the constraint is not satisfied. The relaxed constraint must be marked on the structure such that the type and the position of the detected error can be confirmed later on. As Arabic is a morphologically rich language, various constraints should be met to form a well-formed inflected word, such as imposing certain connected pronouns on specific Arabic verb tense and imposing certain affixes or clitics (e.g. conjunction or pronoun) on specific conjugated verbs. In the proposed Arabic ILTS these two example constraints can be relaxed to allow for error diagnosis.

The following steps are used to get base word forms from all words in the learner input:

1. (1) Extract a list of all possible suffixes.

2. (2) Filter the suffix list to exclude irrelevant suffixes using heuristic rules.

3. (3) Extract a list of all possible prefixes.

4. (4) Filter prefix list using heuristic rules.

5. (5) Extract a list of all possible stems.

6. (6) Group identical stems together.

7. (7) Normalize the stem to get base word forms.

In the following we show the application of each of these steps on the word /qAltw/ (I-told), the remaining word from answer (e) of Example 6, to get the base word forms.

Step 1.Extract a list of all possible suffixes

The system uses regular expression to describe the list of affixes.Footnote ¹⁹ The regular expressions are represented as deterministic finite-state automata. The suffix list is expressed in reverse order to efficiently parse the input left to right, i.e. the suffix characters sequence is matched from the leftmost (suffix) positions of the words. Figure 2 illustrates a finite-state automata representation of four suffixes /Teh, Alef, Alef-Teh, Waw-Alef/.

Fig. 2.

A finite-state automata for four suffixes.

To extract the suffix list, the system parses the input word against the suffix automata. The parsing begins at the end of the input word and works backward. The system relaxes the usage of certain affixes constraint by using the three-way-match method to compare two strings (Elmi and Evens Reference Elmi and Evens1998): suffix of learner input with an Arabic proper suffix. This method assumes that when a character at location n in the first string does not match the character at location m in the second string, we have an errorFootnote ²⁰ and two other comparisons are made (character at position n with character at position m +1 and character at position n + 1 with character at position m), where n represents a position at the input string and m represents a position (state) at the finite-state automata. Initially, n = 1 to point the last letter in the input string and m = 0 to point the letter at the initial state (last suffix letter) in the finite-state automata. The three-way-match comparison and the order of the comparison are shown in Figure 3.

Fig. 3.

The three-way-match comparison and the order of the comparison.

For example, given the finite-state automata shown in Figure 2 and the input word * /qAlo-tw/ (I-told), the word analyzer proceeds as follows: (1) match the last letter (n = 1) /w/ of the input word with the Arabic proper suffix /t/ that occurs at the end (m = 1) of Arabic verbs. The match fails. So it tries to match again with the letter but last (n + 1 = 2) of the input word, i.e. letter /t/, which succeeds. This process considers the letter /t/ as a possible suffix and the letter /w/ as an extra letter occurring at the end of the input word. Similarly, the match exhaustively proceeds with other Arabic proper suffixes yielding ten possible solutions as shown in Table 7.

Table 7.

The list of possible suffixes of the input word * /qAlo-tw/ (I-told)

In practice, the use of constraint relaxation technique in the analysis leads to over-generation. In order to resolve this issue and to identify the analyses which best reflect the learner's intention, we applied a set of heuristic rules to eliminate highly implausible analysis. For example, SLLs of Arabic find it difficult to differentiate vowel signs from genuine characters, such as the confusion about the use of the short vowel Damma (

) /u/ versus the long vowel letter

/w/. Therefore, the system restricts the inserted or missing characters to only weak letters. Moreover, the system restricts the converted character to be a character from a local set of similarly pronounced characters. The closely related pronounced characters are grouped into the following sets:

• 

Step 2.Filter suffix list using heuristic rules

The suffix list could be reduced tangibly in a number ways taking into consideration the set of error categories handled by the system. For example, the list of possible suffixes in Table 7 could be minimized to the following five solutions:

• • Suffix solution (1): NULL suffix.

• • Suffix solution (4): First person singular perfect verb suffix with extra Waw.

• • Suffix solution (8): Second person male plural imperative verb suffix with missing Alef.

• • Suffix solution (9): Male plural imperfect verb suffix with missing Alef.

• • Suffix solution (10): Third person male plural perfect verb suffix with missing Alef.

Consequently, five (half of) solutions were removed. There are two reasons for these decisions. Currently, the system does not handle errors related to Arabic nouns, hence led to ignoring the second and the seventh solutions. Nevertheless, the other three solutions (third, fifth, and sixth) are discarded since their end case mismatches the extra character /w/. This error type is the misconception of making a short vowel long vowel.

Step 3.Extract a list of all possible prefixes

Extracting the prefix list is similar to extracting the suffix list except that the order of the match process begins at the first letter and upwards. For example, applying this step on the word /qAlo-tw/ (I-told), we get only a null prefix in the prefix list as shown in Table 8.

Table 8.

The list of possible prefixes of the input word * /qAlo-tw/ (I-told)

Step 4.Filter prefix list using heuristic rules

The prefix list could be minimized according to the set of error categories handled by the system. For example, the system does not handle errors related to Arabic nouns. So any solution related to nouns is just ignored. Applying this step on the word * /qAlo-tw/ (I-told) does not filter the result.

Step 5.Extract all possible stems

In order to extract a possible stem, the system tries to get every possible substring from the input word that remains after relaxing morphological constraints. In this case the constraint under consideration is ‘Imposing certain pronouns on specific verb tense’ (i.e. although the tense feature of the verb does not agree with its prefix and/or suffix, this solution is still counted). Otherwise, i.e. there is no such constraint to be relaxed, the system tries to verify the compatibility rules between a prefix and a suffix from a predefined list of prefixes and suffixes,Footnote ²¹ discarding any disagreement due to incompatible combination.

The output of this step yields four solutions as shown in Table 9. Each solution consists of five elements constituting the breakdown of the word into three segments: prefix, stem, and suffix, an FS that describes the analyzed word, and an initial error hypothesis. The affix is represented by three components: the affix itself, an ID, and the affix's FS. The stem is represented by the actual sequence of Arabic script. The error hypothesis is represented by a structure that indicates the operation (insert, delete, or convert) required to relax the affix and whose elements are the actual characters, the position where the operation should take place, and the affix tag.

Table 9.

The list of word analyses of the input word * /qAlo-tw/ (I-told)

Note that in the working example, the discarded incompatible prefix and suffix combination is the solution ‘Null prefixed masculine plural imperfect verb with deleted Alef in the suffix’ (i.e. the ninth suffix solution in Table 7) because in Arabic morphology it is invalid to use a combination between a null prefix and the imperfect suffix ‘’ (Waw and Alef) and there is no matter of relaxing this incompatibility issue. This suffix ‘’ can only be used with either the prefix ‘’ (Yeh) or ‘’ (Teh).

Step 6.Group identical stems together

The solution list may include similar stem strings. For the sake of efficiency, each group of similar stem strings is combined together to form an element containing the common stem string and a group of pointers to the corresponding elements in the solution list. After applying this step to the four solutions in Table 9, a list of three elements is formed: {(, [1]), (, [2]), (, [3, 4])}.

Step 7.Normalize the stem to get the base word forms

To get all possible base word forms from each stem, the list of Arabic verb patterns is expressed as deterministic finite-state automata. Figure 4 illustrates a finite-state automata of the patterns , , , /faEala, fAEala, tafoE ~ al, tafAEala/. We differentiate between two types of letters forming the pattern (template): radical and variant ones. For example, the pattern /faEala/ has three radicals, while the pattern /tafoE ~ al/ has one variant letter /t/ and three radical letters.

Fig. 4.

A finite-state automata for four Arabic verb patterns: , , , /faEala, fAEala, tafoE ~ al, tafAEala/. The letter inside a square is generic.

The systemFootnote ²² uses the three-way-match method to parse each stem in the solution against the actual verb pattern and relaxes any missing or transformed similar letters. If the parse succeeds, it deletes from the resultant stem any weak or hamza letters, producing the base word form.

The parsing is applied to the following stem list: {(, [1]), (, [2]), (, [3, 4])}. The parsing of the first solution is not applicable because there is no Arabic pattern that might match. The parsing of the second solution yields intermediate results { and }, which after removing the weak letter from the first stem become the same base form {}. The parsing of the third solution yields the intermediate results { and }, which after removing the weak letter would become the base form {}. Hence, the final solution of word analysis yields the base forms {(, [2]), (, [3, 4])}, which correspond to the following analyses:

• • Null prefix, first person singular perfect verb in active voice, and extra suffix Waw.

• • Null prefix, second person masculine plural imperative verb, and deleted suffix Alef.

• • Null prefix, third person masculine plural perfect verb in active voice, and deleted suffix Alef.

7.2.3 Lexical error analyzer

The lexical error analyzer is the most elaborate of the modules that needs deep Arabic natural language analysis and generation. Heuristics are applied to get rid of irrelevant solutions. Figure 5 presents its architecture. This section explains the components and function of the lexical error analyzer.

Fig. 5.

Architecture of lexical analyzer.

Morphological analysis and generation: The final analysis of every word in the learner's answer is computed by morphological analysis and generation tasks. The morphological analysis is a compound task that takes input from two main NLP tools. The first tool produces a list containing all possible word breakdowns as explained in the previous section. This tool accepts either well-formed or ill-formed Arabic words. The second tool is the online Xerox morphological analyzer, which, like most Arabic morphological analyzers, accepts only well-formed Arabic words, producing all possible analyses of the words having the same root as the correct answer. The decision of whether to accept the stem as a correct analysis of the input word depends on the success of morphologically generating this stem backwards from the root in this analysis using its intended features as specified in the answer associated with the authored question.

The following steps are used to get the final word analysis:

1. (1) Combine the word analysis with the morphological analyzer output

2. (2) Filter the solution list using heuristic rules.

3. (3) Apply morphological generation

4. (4) Compare the generated stem against the analyzed stem

5. (5) Suggest the final word analysis using heuristic rules

Step 1.Combine the word analysis with the Xerox morphological analysis

The main purpose of this step is to form rich morphological representations by combining the results of both word analyzer and Xerox morphological analyzer. Xerox morphological analyzer is considered to be one of the most respected tools that we rely upon for performing morphological analysis of well-formed words, which might form a wrong answer. It is obvious that a leaner incorrect answer might include either well-formed or ill-formed erroneous input, and this was the reason that we developed our word analysis tool. In the example that we use for illustration, the morphological analyzer does not produce any solution.Footnote ²³ Hence, the combined analyses of the ill-formed word * /qAlo-tw/ (I-told) is the solution produced by the word analyzer, which is ‘first person singular perfect verb in active voice with extra Waw in the suffix’

Step 2.Filter the solution list using heuristic rules

The solution list could be improved if we consider a number of factors: learner answer, question parameters, and error categories handled by the system. We developed a set of heuristic rules that excludes irrelevant solutions. Two examples of these rules are presented in Table 10. Considering the illustrative example, none of the heuristic rules were applicable such that no filtering took place to rule out any irrelevant solution.

Table 10.

Examples of filtering rules

Step 3.Apply morphological generation

At this step the morphological generation is done as a reverse process using the analyzed structure. In particular, the Arabic morphological generation tool applies features of every so far analysis to the correct root to generate a well-formed Arabic stem. We followed the same morphological generation approach proposed by Cavalli-Sforza et al. (Reference Cavalli-Sforza, Soudi and Mitamura2000) and Soudi et al. (Reference Soudi, Cavalli-Sforza and Jamari2001) to develop an Arabic shallow morphological generator. This approach has the advantage of reducing the complexity of Arabic morphological generation by employing discrimination tree representation augmented with transformational rules. The morphological generation approach solves the problem of generating a large number of Arabic verbal variants by decoupling the problem of stem changes (infixation) from that of prefixes and suffixes, which has been proved to significantly reduce the number of rules required. A tree is used to build a hierarchical representation that combines the classification of morphological variants. Each internal node of the tree specifies a piece of the feature structure that is common to that entire subtree. The root of the tree is a special node that simply binds all subtrees together. The leaf nodes of the tree correspond to distinct morphological forms in the language. Each node in the tree below the root is built by specifying the parent of the node and the conjunction or disjunction of feature-value pairs (e.g. voice: active, type: hollow) that define the node.Footnote ²⁴ A morphological transformation rule is attached to each leaf node. The rule consists of one or more mutually exclusive clauses, each of which has two parts. The first part of a clause is matched against the value of the root feature. The second part includes one or more operators that will be applied in the given order. Operators include addition, deletion, and replacement of infixes. For a specified feature structure, its features are matched against the features defining each subtree (e.g. perfect: verb) following a path until a leaf is reached. The output of applying transformations rule at the leaf is the transformed stem from its root and other features. An example of rules showing the stem change for perfect hollow verbs of the pattern /faEala/ is shown in Figure 6.

Fig. 6.

A subtree showing stem change for perfect hollow verbs of pattern /faEala/.

In Figure 6, consider the perfect hollow verbs in active voice, where the stem keeps the long vowel

/A/ as a middle letter, for third person singular (e.g.

‘he/it said’ and

‘she-said’), dual (e.g.

‘they-m-dual-said’ and

‘they-f-dual-said’), and plural masculine (e.g.

‘they-m-pl-said’). Whereas, in the remaining person–number–gender combinations the stem takes a short vowel whose diacritics depend on the underlying root of the verb. The transformation rule attached to the leaf node performs the conversion of the vowel (short or long) on the root. For example, applying the morphological generation step on the illustrative example solution, i.e. ‘first person singular perfect verb active voice with extra Waw in the suffix’, the rule deleteMiddleWeakLetter is triggered. This rule converts the weak middle letter in the root

into a short vowel and generates the stem

/qul/, see the following FS:

• 

Step 4.Compare the generated stem against the analyzed stem

In this step, a well-focused three-way-match method is used to parse the generated stem against the analyzed stem. The insertion and deletion of letters take into consideration the weak letters and the conversion of letters considers those that have similar pronunciation. The matching process works as follows: Partition the two stem strings: generated stem = xuz and extracted stem = xvz, where x is the leading segment, z is the trail segment, and u and v are the error segments. First, the leading segment is selected, which might be empty if the leading characters of generated stem and extracted stem are different. In an exact match between these two stems, this segment should contain the whole word. Second, the trail segment is selected, which might be empty if the last characters of both stems are different. Finally, the error segments are the remaining characters of the two stems. The three-way-match method is applied on these two segments to identify any missing, extra, and converted characters.

Consider current solution of answer (e) in Example 6, the analyzed stem is /qAl/ and the generated stem is /qul/. Applying this step on these stems yields the leading segment {}/q/ and the tail segment {} /l/. Hence, the error segment is the middle character {} /A/, which is missing for the generated stem /qul/. Therefore, the system identifies an extra character /A/ in /qAl/. This result is represented by the structure insert(‘’, 2,stem)that indicates that there is extra character(s) in the stem. Table 11 shows the updated error indication.

Table 11.

The word analysis of the word /qAlo-tw/ (I-told) within applying the error analysis

Step 5. Suggest the final word analysis using heuristic rules

Ambiguity is a major problem in NLP. In our case we are exposed to one-to-many ambiguity where multiple interpretations are observed for ill-formed to well-formed transformations. In the proposed Arabic ILTS, relaxing the linguistic constraints in order to analyze a learner answer generally produces more interpretations than systems designed specifically for handling well-formed input. Consider, for example, the erroneous word * / Eiy$ tu /. This word has two interpretations for error corrections: (1) /Ei$ tu/ (I-lived), which is related to problems with vowel letters, confusion that led to using of long vowel /y/ instead of short vowel /i/ “, or (2) /Eay ~ a$ tu/ (I-sustained), which is related to problems with verb pattern, confusion that led to applying the pattern /faE ~ al/ instead of /faEala/.

In general, NLP approaches usually resolve the ambiguity problem by incorporating disambiguation techniques that suggest a preferred solution. In Arabic ILTS, the preferred method is pedagogical and more closely follows the approach used by the language teacher, who considers the likelihood of an error. The likelihood of an error takes into account the level of instruction and the frequency and/or difficulty of the taught Arabic concepts. Our main concern here is to avoid misleading or incorrect feedback. In Arabic ILTS, we decided to handle three types of ambiguous verb analysis that are related to erroneous learner input:

1. (1) When the Arabic verb conjugation imposed by the passive voice and imperative tense has the same orthographic verb form imposed by the active voice and imperfect tense, respectively. The learning skill level can be used to disambiguate this type. Note that it is doubtful that a beginner student of Arabic writes a passive voice verb instead of an active because the passive voice is a rare construction, even at a more advanced level.

2. (2) When a verb affix has different feature-value pair interpretations. For example, the suffix /Teh/ can be interpreted in four ways: (1) first person singular, (2) second person singular masculine, (3) second person singular feminine, and (4) third person singular feminine perfect verb suffixes.

3. (3) When verb patterns share the same orthographic form but differ in vocalization. For example, verb conjugations from the root following the pattern /faEala/ (for a verb indicating did) and /faE ~ al/ (for a verb indicating activated) are different.

In the reset of this section we will show how Arabic ILTS deals with these types of ambiguity. In case of the first type of ambiguity, the system selects the word analysis that a student most likely intended. It implements two prioritized conditions to selects the most preferred word analysis. The first condition: If the question goal is to test passive voice, then the system selects passive voice analysis, otherwise it selects the active voice analysis. The other condition: If the question goal is to test imperative tense, then the system selects imperative tense analysis, otherwise it selects the perfect or imperfect tense analysis. For example, given the following learner input:

1. a. * /ta-biAE jad ~ atiy Al > aruz ~ / (My-grandmother sells the-rice).

2. b. /ta-biyE jad ~ atiy Al > aruz ~ / (My-grandmother sells the-rice).

The learner has incorrectly conjugated a hollow verb in the imperfect tense. The error analyzer produces two possible analyses: third person singular feminine imperfect active verb and third person singular feminine imperfect passive verb. Given the question goal that tests conjugation and usage of imperfect verb active voice, the prioritized condition applies that selects the active voice analysis.

In case of the second ambiguity type, the system merges all of the affix analyses in one structure. For example, given the following learner input:

1. a. /muHam ~ ad tawar ~ aTt fiy jariymap qatol/ (Mohammed was-involved in murder crime).

2. b. /muHam ~ ad ta-war ~ aTa fiy jariymap qatol/ (Mohammed was-involved in murder crime).

The learner here has made a subject–verb disagreement between (Mohammed) the inchoative () of the sentence and its enunciative () . The error analyzer produces four possible analyses, as shown in Table 12. This step will merge the four analyses into one generic solution that combines the structure of affixes, as shown in Table 13. Across the whole analyses, common (identical) feature-value pairs, e.g. subj number: singular, will have an exact instance in the new structure and copied as the same. Whereas fully covered values, e.g. subject gender: masculine, feminine, or neutral, will have a do-not-care value in the new structure indicated by the special symbol ‘$ _’, i.e. subject gender: $ _.

Table 12.

Four analyses of the word /tawar ~ aTt/ due to one-to-many ambiguity of the suffix /Teh/

Table 13.

Resolving suffix ambiguity by merging closely related solutions in Table 12

In case of the third type of ambiguity, the system merges all of the stem analyses in one structure. For example, given the following learner input:

1. a. / jad ~ iy wajad ~ apiy naq ~ aluwA < ilaY bayot jadiyd / (my-grandfather and my-grandmother moved to a new house).

2. b. /jad ~ iy wajad ~ apiy {inotaqalA < ilaY bayot jadiyd/ (my-grandfather and my-grandmother moved to a new house).

The learner here has made two errors: (1) subject–verb disagreement between the inchoative () of the sentence and its enunciative () , and (2) incorrect conjugation of a perfect verb that follows the pattern ‘’ /faEala/ instead of the pattern ‘’ /faE ~ al/. The error analyzer produces four possible analyses as shown in Table 14. This step will merge the two analyses into one generic solution that combines the structure of stems as shown in Table 15. The only difference is the use of do-not-care value with the pattern feature.

Table 14.

Two analyses of the word /naq ~ aluwA/ (moved) due to one-to-many ambiguity of the pattern orthography

Table 15.

Resolving suffix ambiguity by merging solutions in Table 14

Error Detection: This module identifies different error types from the word analysis structure. It relies on production rules to recognize these error types. Figure 7 shows some examples of these rules.

Fig. 7.

Examples of rule-based error detection knowledge.

Note that the learner might make multiple errors in his/her input. So this module exhaustively tests all rules to detect all possible error types the learner has made. Accordingly, it should also update the student model by incrementing the EFC variable for only error knowledge associated with the error made by the learner. This is done by searching the student model database to get the error knowledge index associated with the error made by the learner and then increment the EFC variable associated with this index.

Applying these rules on the word analysis of /qAlo-tw/ (I-told), i.e. answer (e) in Example 6 shown in Table 11, the system detects the following errors:

1. (1) Verb tense error since the tense of the correct verb is imperfect whereas the tense of the analyzed verb is perfect.

2. (2) Long vowel instead of short vowel error since there is an extra suffix character.

3. (3) Verb conjugation error since there is an extra character at position 2 in the stem.

These three errors with their parameters will be sent to the error scheduler module.

Error scheduler: The last step in analyzing learner's answer is handled by the error scheduler module. The task here is to accommodate multiple errors, instructional feedback messages need to be prioritized by the system and displayed one at a time to the student to avoid multiple error reports. While it is desirable to construct a system capable of detecting and accurately explaining all errors, it does not follow that the system should display each and every error detected.

The proposed Arabic ILTS maintains an error priority queue which determines the order in which instructional feedback messages are displayed to the learner. It ranks instructional feedback with respect to the dependency of errors, e.g. verb tense error has higher priority than verb conjugation error.

Applying this module on the word /qAlo-tw/ (I-told), the error priority queue gives a higher priority to verb tense error over the other two errors (i.e. verb conjugation, long vowel error instead of short vowel error). Finally, this error type (verb tense) as well as its parameters ([(‘imperfect’, ‘active’, ‘indicative’) (‘perfect’, ‘active’, ““)]) is to be sent to tutoring module.

8.1 Data collection and error tagging

In the absence of a complete computationally erroneous learner corpus for Arabic that can be used to evaluate the proposed approach and techniques, we have to manually collect the test set from real teaching environment. This corpus should be manually annotated with learner errors information. A real test set that consists of 116 real Arabic sentences is acquired. These sentences are collected from two different sources (i.e. different teaching environments with different Arabic SLLs background): (1) the Arabic Language Institute, at the American University in Cairo (AUC), where Arabic is taught to different learners from different countries, and (2) PhD dissertation that examines Arabic verb errors made by Malay's SLLs of Arabic (Jassem Reference Jassem2000). The number of words per sentence in the whole test set varies from three to fifteen, with an average of 5.1 words per test sentence. The total number of words in all test sentences is 587, of these 118 words have lexical verb errors. Moreover, thirteen verbs fall under multiple lexical errors; eleven out of these thirteen have two errors whereas only two verbs have three errors.

Creating a large annotated learner corpus as a test dataset is expensive and beyond the scope of this research project. It is much related to Corpus Linguistics that has research methods to study data collection of representative elements agreeing on tag set, manual annotation, inter-annotator agreement, verification, and so on. Although the acquired test set is relatively small, we have shown that it is sufficient to demonstrate that the approach and techniques employed in this paper have successfully generated all possible analyses of ill-formed verbs written by SLLs of Arabic, in particular, when it comes to difficult constructions such as Arabic weak verbs.

After collecting this test set, the raw text of the test set is judged by Arabic human specialist who manually identified and classified lexical errors. These errors are annotated in the test set; 133 errors were identified and classified into nine error types that belong to one hundred sources of errors. Figure 8 shows an example of annotated reference set used in evaluation. For clarity purposes, the annotation is presented using XML format.

Fig. 8.

Example of error annotation in XML format.

8.2 The experiment

The unannotated test set is fed into the system and the output (i.e. error-specific feedback messages that conform to the learner expertise level) is compared against the manually annotated reference set. The same annotator who performed the test set annotation also compares the reference set with the system's output. Note that our system provides feedback that conforms to the learning level. So the feedback to the beginner learning level is compared with the annotated source of error in the reference set, while the feedback to the intermediate learning level is compared with the annotated error type. The feedback to the advanced learning level is compared with the annotated class of error.

The system's performance is measured in terms of recall rate for each error type and the precision ratio of the system; a high recall rate indicates that a large percentage of errors have been identified, while a high precision ratio confirms that the system detects real errors and does not overflag much. Note, however, the following four calculations should be made for each error type handled by the system: (1) A precise ratio of the number of errors in the test set; (2) the number of errors which has been successfully diagnosed by the system (i.e. the source of error is identified), (3) the number and types of errors which the system has successfully detected but failed to identify their exact source (i.e. partially diagnosed errors). For example, the system might detect subject–verb disagreement but fails to identify the source of this error such as disagreement in person, gender, and number, and (4) the number of errors which have been successfully detected but failed to identify their types (i.e. general error indication).

8.3 Analysis of results

Table 16 shows experimental results. The first column describes different error types handled by the system, while the second column presents the frequency of each error type in the test set. The rest of columns present the recall rate of complete diagnosis, partial diagnosis, and general error indication, respectively. Note, however, that in the case of partially diagnosed errors, the system produces the same error-specific feedback message. The reason is that the system was not able to detect the source of error, i.e. only the error type is detected. The feedback message in the case of general error indication is a catch-all error message for all learning levels (there is an error in the word . . .). The recall rate metric is used to measure the system performance. The recall rate is shown in the last raw. It is calculated by summing up all the diagnosed errors for each of the last three columns and then dividing each of them by the total number of errors (133 errors) in the test set, e.g. 99/133. The precision ratio is 100 percent (i.e. 99/99), it is calculated by summing the number of correctly diagnosed errors by the system, and then dividing it by the total number of errors diagnosed by the system, including the cases of erroneously diagnosed errors (i.e. overflagging or overdetection cases). Note, however, that our system is designed such that it never overflags, since the first step in the system is to match correct answer against the learner answer such that it does not identify correct input by the learner as an error.

Table 16.

Results of applying the proposed Arabic ILTS on real test set

Following can be observed in Table 16. The highest frequent errors made by Arabic SLLs were vowel letters and the subject–verb disagreement with a frequency of 24 for each of them. The accumulated frequency of these two errors together was 48 out of 133, which represents 36.09 percent of total errors. The least frequent error is verb mood, which was found twice. It represents 1.5 percent of the total errors found.

The highest recall rate was for both consonant letters and verb mood that achieved 100 percent. The least recall rate was for verb pattern (57.14 percent). This arose because of the errors ambiguity problem, the system has no direct knowledge of what the student meant to express. For example, when the learner's answer includes the word instead of /ta-Eal ~ am-tu/ (I-study), it is not clear whether the learner meant /Ealim-tu/ (I-knew) that has the pattern /faEil/ or /Eal ~ am-tu/ (I-taught) that has the pattern /faE ~ al/. In this case, the system was able to successfully detect that the error type is verb pattern, but could not identify the wrong pattern. Moreover, the general error indication also arose from ambiguous situations. For example, the word can have four possible interpretations: the verb / > a-komul/ (I-become complete), the verb / > u-komil/ (I-complete), the verb / > akomala/ (he/it completed), and the noun / > akomal/ (perfect).