3.1 Techniques in native language identification

The first feature-based method (Tsur and Rappoport Reference Tsur and Rappoport2007) found that incredibly simple top 200 frequent bigram character features fed to SVM led to 66 per cent accuracy on the five native languages. They claimed that word choice of non-native speakers is influenced by the phonology of their native language (as evidenced by the effectiveness of the character features). This approach is compared to a unigram words baseline which achieved only 47 per cent accuracy. They finally hypothesized that using a spoken-language corpus would achieve even stronger results favoring character bigrams. Their reasoning was that much less conscious effort is put into speaking than writing. For analyzing transcripts of spoken words, the ICNALE corpus may be applicable, described in Section 3.

NLI has also been approached through contrastive analysis (Wong and Dras Reference Wong and Dras2009): the idea that errors in text are influenced by the native language of the author. They investigated three error types as features: subject-verb disagreement, noun-number disagreement, and determiner misuse. These error types are then used as ‘stylistic markers’ for NLI features with an SVM classifier. To find these errors in text, they used an open source grammar checkerFootnote ¹⁰, as opposed to professionally edited text. Interestingly, ANOVA showed that the features had a measurable effect, but after combining their contrastive features with existing methods, they were not able to significantly increase the classification accuracy from Koppel et al. (Reference Koppel, Schler and Zigdon2005).

The previous authors (Wong and Dras Reference Wong and Dras2011) follow their work on contrastive analysis, attempting to amend its shortcomings. Instead of error types, they use two different features obtained from grammatical parse trees: horizontal slices (production rules) and parse rerankings. They claim these are the first pure syntactic features used in NLI. For the production rules, they immediately applied information gain dimensionality reduction. The reranking features are those contained in the Charniak parserFootnote ¹¹ and Stanford ParserFootnote ¹² trained on the Wall Street Journal. Unlike the previous two attempts, the authors found MaxEnt to outperform SVM as the classifier. Additionally, five-fold cross validation was performed (as opposed to ten-fold), which means the accuracies cannot be precisely compared with previous work. In any event, they report a final accuracy of 80 per cent, which was the highest reported as of 2012.

Wong, Dras and Johnson (Reference Wong, Dras and Johnson2012) explore the last author’s—Mark Johnson’s—adaptor grammars (Johnson, Griffiths and Goldwater Reference Johnson, Griffiths and Goldwater2006) to generate features. Simply, adaptor grammars are a non-parametric extension to PCFGs (probabilistic context free grammars). They can learn arbitrary-length word sequences (collocations); for example, gradient descent and cost function were learned as phrases in a machine learning topic. These adaptor grammars are used in two ways: in the first, collocations are used as features in a MaxEnt classifier. In the second, the grammar is trained on each class (representing native language). At test time, the most probable grammar to have generated the text is selected. For both tasks, the authors use five-fold cross validation on seven native languages. In the feature-based classification, they achieved 76 per cent; in the language model-based classification, they achieved only 50 per cent, a performance similar to the unigram word baseline from Tsur and Rappoport (Reference Tsur and Rappoport2007).

Swanson and Charniak (Reference Swanson and Charniak2012) made use of tree substitution grammars (TSGs) (Blunsom and Cohn Reference Blunsom and Cohn2010). TSGs are a tree-rewriting formalism that defines operations on partial (parse) tree objects. For example, subtrees may be added or removed from a base tree. Benefits of using this method are priors which prefer smaller production rules and the ability to capture long-range dependencies. Various induction methods are compared to generate features, and five-fold cross validation on seven native languages is performed. All TSG features outperformed the CFG baseline (at 73 per cent). The highest TSG induction method was Bayesian induction at 78 per cent.

Massung, Zhai and Hockenmaier (Reference Massung, Zhai and Hockenmaier2013) also make use of grammatical parse tree features, but mainly focus on their structural aspects as opposed to the syntactic category labels. In one classification task, they found these features to work well in determining the nationality of student essay writers from the CEEAUS dataset. These structural parse tree features may be applicable in other tree-based objects such as adaptor grammars, TSGs, and dependency parses, but this has yet to be explored.

Lastly, Stolerman, Caliskan and Greenstadt (Reference Stolerman, Caliskan and Greenstadt2013) use language family as a feature for NLI. Specifically, they took ICLEv2 documents classified with low confidence using unigram features and reclassified them into one of the Germanic, Slavic, or Romance language families. From there, the document was classified as an L1 using a subset of training data only from the language family it was classified as. This method is a type of hierarchical classification problem, and achieved about a 6 per cent increase over the baseline in NLI accuracy when using information gain feature selection.

In summary, Table 2 lists the comparable accuracies from experiments run on the ICLE/ICLEv2 subset of five European languages. In general though, accuracies between 70 per cent to 80 per cent are standard for a wide variety of techniques and corpora.

3.2 Shared task in native language identification

The eighth Building Educational Applications workshop held a shared task dedicated to NLI (Tetreault, Blanchard, and Cahill Reference Tetreault, Blanchard and Cahill2013). Its goal was to increase visibility of the problem and standardize results on a larger dataset: ETS. In total, twenty-nine teams participated in the shared task.

There were three separate subtasks, differentiated by what data was allowed for training:

• • Closed Training: Use only the ETS corpus for training. All twenty-nine teams attempted this task.

• • Open Training I: Use any data except the ETS corpus for training; the evaluation is still done on the ETS test set. Only three teams attempted this task.

• • Open Training II: Use any data including the ETS corpus. Only four teams attempted this task.

Overall results for the closed training were relatively close; the top thirteen teams had between 80–83.6 per cent accuracy. The open training I subtask teams performed much lower (probably as expected) at around 33 per cent, 38 per cent, and 57 per cent accuracy. Finally, in the open training II subtask, none of the four teams were able to beat the highest closed score. The most useful results for evaluating models are probably those of the closed task, since all training data is uniform, though it is interesting to see approaches employed to increase the amount of training data. For example, in the open subtasks, other NLI datasets were used such as ICLE, FCE, and ICNALE (all described in Section 2). Additionally, some teams crawled Lang-8, also described in Section 2.

The most common features used were n-grams of lexical tokens such as words or part-of-speech tags; Figure 1 compares the n-grams used by all teams. Aside from these, some less common features were grammatical: dependency parses, TSGs (Blunsom and Cohn Reference Blunsom and Cohn2010), parse tree rules, and adaptor grammars (Johnson et al. Reference Johnson, Griffiths and Goldwater2006) were used. Spelling error features were also captured by three of the teams. These features are all described in work in Section 3.1. Below, we outline a few of the more unique feature representations.

Fig. 1.

Number of teams using particular n-gram features (left) and number of teams using a particular machine learning algorithm (right). DFA and PPM refer to discriminant function analysis and prediction by partial matching.

Skipgrams (Guthrie et al. Reference Guthrie, Allison, Liu, Guthrie and Wilks2006) were an n-gram variant feature used by several teams. Instead of considering an n-gram to consist of n adjacent words in the text, a k-skip n-gram allows up to k words total to be skipped in a sequence of tokens. Given the sentence ‘They all studied statistics Monday evening’, normal 3-grams would be {They all studied, all studied statistics, studied statistics Monday, statistics Monday evening}. A few possible 2-skip 3-grams are {They all studied, They studied statistics, They statistics Monday}. This attempts to capture important phrases without regard to interspersed function words. Using skipgrams was shown to reduce perplexity on a testing set, as well as offer a viable alternative to increase the corpus size (which is usually not feasible). Since most NLI datasets we have examined are relatively small, using skipgrams could help improve language modeling. While interesting, none of the teams in the top third used skipgrams in their systems.

For one feature type, the system by LIMSI used a form of machine translation called ‘back translations’ (Lavergne et al. Reference Lavergne, Illouz, Max and Nagata2013). A back translation attempts to capture a writer’s lexical preference for a word sense. For example, LIMSI found that the English word sense for awkward is more likely to be written as clumsy by native Spanish speakers. These word preferences were used as features in addition to other features: word n-grams, spelling mistakes, and grammatical mistakes. Adding the back translation features slightly increased the task’s overall classification accuracy, but the team still performed poorly overall.

The CMU-Haifa team used a ratio of passive to active verbs in their system (Tsvetkov et al. Reference Tsvetkov, Twitto, Schneider, Ordan, Faruqui, Chahuneau, Wintner and Dyer2013). They operated under the common assumption that English uses passive voice more frequently than other languages, and the amount of passive use may vary based on the writer’s L1. Recall that passive voice is a literary technique that shifts attention away from the one performing an action. ‘The passive voice is often used’ is a sentence in the passive voice; ‘English speakers often use the passive voice’ is a sentence in the active voice. The passive voice is characterized by the verb to be and the past participle of the verb. In the previous sentences, we have is used and use conjugated differently. Using the passive ratio feature exclusively yielded a 12 per cent accuracy on a 9 per cent baseline. In combination with four other main features, the accuracy was not significantly improved. However, further investigation could be warranted to examine the usefulness of such stylistic composition features.

SVM was by far the most prevalent classifier employed by the teams, as displayed in Figure 1. Below, we outline some of the more uncommon classifiers.

Discriminant function analysis from statistics was used for increased interpretability over other methods (Kyle et al. Reference Kyle, Crossley, Dai and McNamara2013). Using this model, they were able to find correlations between different L1s. For instance, Japanese and Korean L1s were highly correlated in their underuse of the words {all, any, but, different, or, person, this, your}. Unsurprisingly, languages of similar origin were also correlated—the Romance languages were often misclassified as one another due to increased correlation of n-gram tokens. Unfortunately, this team performed poorly overall.

Originally designed to operate on DNA sequences, the string kernel model (Popescu and Ionescu Reference Popescu and Ionescu2013) is given a stream of characters. Specifically, a kernel based on local rank distance is used for NLI. The simplest string kernel function f(w ₁, w ₂) counts the number of substrings of a particular length that w ₁ and w ₂ share. It is also quite intuitive to introduce a normalized version that is not biased by long words. Local rank distance is then an extension that counts the sets of similar n-grams between w ₁, w ₂. In their NLI context, each word was actually a character n-gram. This method has the advantage that no syntactic information is needed; no parsing or even sentence or word segmentation is required. In training, they found that n ∈ [5, 8] gave the best results, allowing them to take third place overall in the closed task.

Similar to the string kernels previously, Bobicev (Reference Bobicev2013) uses a method which requires very little text processing: prediction by partial matching, a statistical compression method. The output of prediction by partial matching is a language model which can be used to find the highest likelihood L1 given some unknown text. Both words and characters were used as tokens, though character features performed much worse than words, obtaining a precision of 37 per cent (compared to 70 per cent from the word tokens) on the 9,900-document training set. This performance placed the prediction by partial matching features in the bottom third of contestants.

In conclusion, most methods used by teams in the NLI shared tasks were very standard (e.g. n-grams of words with SVM). In fact, the winning team (Jarvis, Bestgen and Pepper Reference Jarvis, Bestgen and Pepper2013) with 83.6 per cent accuracy used SVM with unigram to trigrams of log-entropy weighted lexemes, lemmas, and POS tags. A small number of teams investigated some unique text representations and classification techniques. Given options between feature-based and likelihood-based strategies, most teams focused on features. While not always successful, these new methods can be further explored and analyzed in future work. Table 3 summarizes the results discussed on the ETS dataset. After the contest ended, others have surpassed the highest-scoring Jarvis et al. team.

Table 3.

Increasing state-of-the-art NLI accuracy results for the ETS dataset

First, Bykh and Meurers (Reference Bykh and Meurers2014) considered an ensemble of various features, consisting of context-free grammar rules and forty n-gram representations of words, POS tags, and lemmas for n ∈ [1, 10]. They found training a separate classifier for each feature type gave definite advantages over concatenating feature vectors.

Second, Ionescu, Popescu and Cahill (Reference Ionescu, Popescu and Cahill2014) improved over Bykh and Meurers (Reference Bykh and Meurers2014) by using only character-based features. However, the way that the character features are used is advanced. They combined various string kernels with two learning methods, kernel ridge regression and kernel discriminant analysis. We suggest that the reader consult their paper for a more in-depth description of these tools. Their best results were when string lengths were in the range [5, 8]. Seven of their kernel and learning combinations outperformed the first place shared task team (Jarvis et al. Reference Jarvis, Bestgen and Pepper2013), and their best two combinations best the Bykh and Meurers score. Their highest-scoring setup was kernel discriminant analysis on a weighted combination of a presence bits kernel (dot product of occurring characters) and the intersection string kernel.

4.1 Techniques in non-native grammar correction

Some grammar correction methods are targeted towards a very specific subset of errors, often categorized by the corpus; others attempt to solve more general errors concerning word sense or collocations. Evaluation for grammar correction is much more varied and unstandardized in comparison to the configurations from NLI seen in the previous section. Which non-native corpus is used also dictates the types of errors that can be corrected. Table 4 compares the different methods discussed in this section and categorizes them as either targeted or general approaches.

Table 4.

Comparison of grammatical error correction strategies. * indicates targeted approaches and ⁺ indicates general approaches

Lee and Seneff (Reference Lee and Seneff2006) train a trigram language model on a lattice of alternatives, where ‘alternatives’ are prepositions, articles, and auxiliaries that may or may not occur between words in the original text. For example, the sentence I want flight Monday can be corrected by inserting two tokens as such: I want a flight on Monday. Their algorithm first strips all such alternatives from the original sentence. So far, this is not much different from the article and preposition corrections. However, they additionally change each remaining word in the input sentence to be a set of related words to the base form: want → {want, wants, wanted, wanting}. Their language model then outputs the k-best candidates. Next, these candidates are given to a PCFG and reranked. The final output is the top-ranked sentence from the PCFG. Across all experiments, they found that reranking the language model candidates significantly increased the F measure.

West, Park and Levy (Reference West, Park and Levy2011) use bilingual random walks between L1 and L2 word senses. For example, on one side of a bipartite graph are L1 words. There are connections from a word w ∈ L1 to a word w′ ∈ L2 if a w could be translated into w′. w could be the English word head, and be translated into a physical head, head of an organization, or the verb to head. This model was used to correct non-native sounding phrases such as entire stranger to the more natural complete stranger. This bipartite graph was combined with a language model to correct non-native sentences. In these experiments, the native language was Korean. Evaluation was performed with Amazon Mechanical TurkFootnote ¹³ where workers chose between the corrected sentence and the original sentence. Results were not strongly positive, since sometimes the corrected errors changed the meaning of the sentence or made it ungrammatical. In future work, the authors suggest using a richer probabilistic model such as a PCFG.

Dahlmeier and Ng (Reference Dahlmeier and Ng2011a) use the NUCLE corpus to find and correct collocation errors via machine translation. Here, a collocation is a phrase commonly used by native speakers. The authors propose that when a writer mentally translates from L1 to L2, some unnatural phrases result due to word choice. They give an example, ‘I like to look movies’ that might be written by a native Chinese speaker since watch and look are very similar in the L1. It would be possible to correct this to the more grammatical ‘I like to look at movies’, but it still does not sound natural. Instead, look is replaced by watch, resulting in the more fluent collocation watch movies. For their experiments, they assume the unnatural collocations have already been identified; this mimics a system where a user may ask for improvement suggestions for a snippet of writing. They train a statistical machine translation model on a parallel Chinese–English corpus to correct collocation errors in the NUCLE corpus. A log-linear model was used to score the candidate phrases which allows additional spelling, homophone, and synonym features to be incorporated. They evaluated their method as a retrieval task, where they returned the top k suggestions to fix each collocation error. Two native-English speakers judged results from 500 corrections with good rater agreement. Finally, they performed an analysis of errors and found that the main reason top-ranked phrases were not correct was due to out-of-vocabulary words.

Dahlmeier and Ng (Reference Dahlmeier and Ng2011b) introduce an alternating structure optimization approach to GEC. In short, alternating structure optimization is able to leverage a common structure between multiple related problems; see Ando and Zhang (Reference Ando and Zhang2005) for a more detailed description. In this case, the related problems are selection (find features from native text) and correction (fix the errors in non-native text). Targets were article and preposition errors, again using the NUCLE corpus. It was shown that alternating structure optimization significantly outperformed a simple linear classifier as well as two unnamed commercial grammar checkers. Features included part-of-speech tags, hypernyms from WordNet, named entities, and shallow parsing tags.

Gamon (Reference Gamon2010) combined a language model and classifier into a ‘meta-classifier’ that detected errors in both article and preposition use. Additionally, they investigated how much more training data is needed for the individual methods to approach the accuracy of the meta-classifier. Features for the classifier were a window of six tokens to the right and left of errors, POS tags, and lexical head features. The classifier was actually split into two steps: first, determine the likely presence of a preposition or article. Then, determine which article or preposition should be chosen. The language model is trained on LDC’s Gigaword corpus and log-likelihood was used (normalized by sentence length). The meta-classifier uses features generated from the two primary models such as ratio of likelihood scores from the language model and classifier decisions. As expected, the meta-classifier outperformed the two simpler models. Prepositions were harder to classify than articles and they required more training data to reach a specific level of accuracy. Future work would be including more primary models to feed features into the meta-classifier.

Madnani, Tetreault and Chodorow (Reference Madnani, Tetreault and Chodorow2012) applied ‘round-trip’ machine translation to correct generic errors in L2 text. A round-trip translation used the Google Translate APIFootnote ¹⁴ to translate the candidate text from L2 to eight different languages and back again to L2. Since it is not guaranteed that the translations will preserve the meaning of the sentence, they assert that using a language model to select the most fluent choice is not acceptable. Instead, they combined alignments between the source and each round trip translation to create the final answer. This method had a better likelihood of maintaining the sentence’s original meaning. In order to evaluate their system, they had human graders check whether the fixes were fluent as well as retaining the original meaning.

The next two papers we discuss consider alternatives to standard acquisition of training data for GEC. Usually, a dataset with annotated areas as described from Section 2 is used, but these papers create or acquire their own non-native errors.

Instead of training directly on non-native text, Rozovskaya and Roth (Reference Rozovskaya and Roth2010) took native text and intentionally inserted article errors. In order to create a realistic error distribution, they generated the article errors with the same observed distribution as the non-native corpora. Advantages to training data generation in this style are avoiding expensive data annotation, circumventing small corpus sizes, and tailoring the classifier to a particular language. Most of all, their method was shown to be superior to only training a classifier on purely native data. One final note they make is that previous baselines used in the literature are not always appropriate; baselines mentioned before are simply the majority class. While this is acceptable for a selection task, this ignores the actual error rate in the data. Usually, the error rate is much lower (shown to be about 10 per cent in their experiments) than the majority class. Therefore, they argue, a more fair comparison would be the error reduction, and this is indeed the measure they use to report their results. They reduced the error detection in native Chinese, Czech, and Russian text by 10 per cent, 5 per cent, and 11 per cent respectively.

The next authors (Cahill et al. Reference Cahill, Madnani, Tetreault and Napolitano2013) obtained their grammatical errors from Wikipedia revisions. They filtered the article revisions dataset looking for single-word changes that corresponded to correcting article and preposition errors. Using this method, they obtained over one million corrections. They compared their mined corpus with standard error correction corpora as well as artificially-inserted errors like Rozovskaya and Roth’s approach. They found that the larger ‘somewhat clean’ Wikipedia edits were much better in increasing the F ₁ of the grammar corrector as opposed to other common datasets, including a smaller ‘more clean’ version of their Wikipedia data. Furthermore, they found that artificial error insertion methods trained on their Wikipedia revisions data increased system accuracy compared to training on smaller corpora, and even generalized across datasets.

In sharp contrast to NLI evaluation, Chodorow et al. (Reference Chodorow, Dickinson, Israel and Tetreault2012) describe many issues regarding evaluation for GEC, with a focus on non-native sentences. This is a valuable paper for those engaging in any GEC task. Their main issue is the ‘three-way contingency’ between the original sentence, the human correction, and the system output (let alone the evaluation scripts themselves). Mentioned later (Rozovskaya and Roth Reference Rozovskaya and Roth2013), this report also emphasizes the significance of the relatively low error rates in non-native sentences—this phenomenon suggests using more interpretable measures such as true and false positives and negatives. Furthermore, how is the severity of an error taken into account? For example, most native English speakers often misuse who and whom. Their conclusion from these observations is to develop a robust evaluation system based on raw error type counts. This allows bias and error skewness to be perceived by the reader while simultaneously permitting the reader or other evaluator to map the raw error data into another form such as precision, accuracy, prevalence, or bias.

4.2 Shared task in non-native grammar correction

We continue our discussion on GEC with the introduction of the CoNLL-2013 shared task (Ng et al. Reference Ng, Wu, Wu, Hadiwinoto and Tetreault2013) and the CoNLL-2014 shared task (Ng et al. Reference Ng, Wu, Briscoe, Hadiwinoto, Sustano and Bryant2014). The training data was the NUCLE corpus, containing annotations categorized by five error types in 2013 along with the corrected text. In 2014, there were a total of twenty-eight error types, many of which were subcategories of the 2013 types. Some systems first classified potential errors by error type and then corrected errors while others made no such distinction. Additionally, the teams received preprocessed input: sentence segmentation, word tokenization, and part-of-speech tagging were all performed.

In all, there were 1,397 essays consisting of 57,151 total sentences. Article and determiner errors were most prevalent. Testing data was considerably smaller with only fifty essays each year. Teams were also allowed to use any external resources that were non-proprietary and publicly available. Table 5 shows the methods of the top teams from both years. The five error types from 2013 are listed below:

• • Article or determiner: ‘In late nineteenth century, there was a severe air crash happening at Miami international airport.’ Correction: replace late with the late.

• • Preposition: ‘Also tracking people is very dangerous if it has been controlled by bad men in a not good purpose. Correction: replace in with for.

• • Noun number: ‘I think such powerful device shall not be made easily available.’ Correction: replace device with devices.

• • Verb form: ‘However, it is an achievement as it is an indication that our society is progressed well and people are living in better conditions.’ Correction: replace progressed with progressing.

• • Subject-verb agreement: ‘People still prefers to bear the risk and allow their pets to have maximum freedom.’ Correction: replace prefers with prefer.

Table 5.

Top teams from the CoNLL-2013 and CoNLL-2014 GEC shared task

While the contest setup and evaluation does not completely mimic real-life scenarios, it does provide useful information for those interested in GEC. Since the errors are categorized into groups, it is possible to perform an error analysis on the results.

As mentioned previously, even performing the evaluation is not a trivial task; suggested corrections—while completely valid—may not actually match the ‘official’ corrections. To mitigate these issues, the contest organizers permitted the contestants to submit their own gold standard corrections if they disagreed with the provided answers. Of course, the corrections were reviewed. It is also worth noting that the training and test data contained errors other than those in the five categories, but these additional errors were not taken into consideration in the final scoring.

The 2013 third-place team (Xiang et al. Reference Xiang, Yuan, Zhang, Wang, Zheng and Wei2013) from Harbin Institute of Technology, split their system into two parts: a machine learning module (article vs determiner, prepositions, and noun errors) and a rule-based module (verb form and subject-verb agreement). As indicated in Table 5, the classifier used was a Maximum Entropy classifier with additional feature selection via genetic algorithms and confidence tuning based on the maximum entropy score. The rules were first preprocessed with the frequent pattern mining algorithm FP-growth to find common incorrect phrases. These common phrases were then removed from the candidate set to be corrected. A list of these common phrases was provided and may prove useful for future work. In their results, they found that their models were very sensitive to the parameters and that the confidence tuning had the largest positive effect in their pipeline.

National Tsing Hua University (Kao et al. Reference Kao, Chang, Chiu, Yen, Boisson, Wu and Chang2013) finished in second place in 2013 with a system that was very similar to theirs from the previous year. In the CoNLL-2013 shared task, subject-verb agreement was not a labeled error type so the National Tsing Hua University system ignored these errors. In the seemingly current trend, a module was created for each error type, though their modules were more similar to each other than the previous systems. Each module used a moving window of words up to length five—five here because they leverage the Google 5-gram corpus as well as their previous tool Linggle.Footnote ¹⁵ Linggle is a linguistic search engine in which the user can specify wildcards for nouns, verbs, and adjectives (Boisson et al. Reference Boisson, Kao, Wu, Yen and Chang2013). Results were returned with those wildcards filled in ordered by increasing likelihood in the dataset. These likelihoods were used to score candidates generated by their system in each module. Their candidate voting system was created using a backoff model of lower-order n-grams. Verb-form errors were really the only significantly different module by including pointwise mutual information.

The University of Illinois at Urbana-Champaign (Rozovskaya et al. Reference Rozovskaya, Chang, Sammons and Roth2013) had the best-performing system in 2013. They made use of previous work for article errors (with averaged perceptron) and preposition errors (with Naïve Bayes). The remaining three error classes were dealt with Naïve Bayes with priors trained on the same Google corpus that National Tsing Hua University used. Although the CoNLL corpus already contained part-of-speech tags, they reparsed the dataset with their own tools in addition to running a shallow parser for feature generation. There was no pipeline in the system, meaning that corrections of each model were pooled together to create the final output sentence. In their error analysis, they found that an incorrect verb form can cause parsers to break, which greatly hindered rule-based methods. Since the errors were so sparse, they hypothesize, less-complicated machine learning algorithms would be more robust to the large amount of noise. This analysis provided much insight and would be useful for future work.

After the contest ended, University of Illinois at Urbana-Champaign (Rozovskaya and Roth Reference Rozovskaya and Roth2013) returned to the CoNLL-2013 shared tasks with a joint learning and inference approach. Based on their observations from the shared task, they reasoned that individual (read: independent) classifiers per word do not capture interactions between errors and word choice. They accomplish this by combining individual classifiers using integer linear programming—a model which is able to jointly learn the error occurrences. They thus increased the F ₁ score on the CoNLL-2013 data to 42 per cent, significantly higher than their first place score. They do note though, that F ₁ may not be an appropriate measure; it is indeed more intuitive to measure the increase in correctness of the original data since it is quite likely that the grammar correction systems actually introduce errors themselves.

Cambridge University (Felice et al. Reference Felice, Yuan, Andersen, Yannakoudakis and Kochmar2014) was the second-best team in 2014. Without additional error suggestions for evaluation, they were in first place. They used a pipeline-based system which first ran rule-based corrections with a tool designed for English learners at their school. A language model selected the best candidate sentences from the rule-based edits. Then, those candidates were sent to a statistical machine translation system trained on parallel English data, which included NUCLE and FCE (see Section 2). Then, a large 5-gram language model from Microsoft was used to select candidates for the final stage. This is the main novelty of their system: with these final candidates they filter the error correction types to ignore unnecessary corrections. Their classifier was able to label almost 70 per cent of error types correctly on development data. They filtered out the labeled types that had zero precision, which consisted of reordering errors, run-ons, comma splices, and acronyms. Although the filtering did increase their F _0.5 score, the difference was not statistically significant. This is an example of how evaluation metrics play an integral part in system design.

Columbia University and the University of Illinois at Urbana-Champaign (Rozovskaya et al. Reference Rozovskaya, Chang, Sammons, Roth and Habash2014) was the top team in 2014. They improved upon their system from the previous year, and also included their joint inference work (Rozovskaya and Roth Reference Rozovskaya and Roth2013). The last addition was the inclusion of new classifiers for the different error types; in total they had sixteen classifiers targeting the twenty-eight error types. The classifier learning algorithm for each error type was one of Naïve Bayes, Naïve Bayes with priors, and averaged perceptron.

In conclusion, we saw a variety of techniques for correcting grammatical errors, which can be categorized into targeted vs general strategies. Targeted strategies focus on errors of only specific types (such as the shared task) which general correctors try to improve the overall fluency of the L2 text.