3.1.1 Distinct

The choices in a well-formed multiple choice question should be mutually exclusive, and only one choice should be correct (Haladyna et al. Reference Haladyna, Downing and Rodriguez2002). DQGen 2012 constrained all the distractors to differ both from the answer and from each other. However, one of the questions in the pilot study included both ‘throat’ and ‘throats,’ which differ but share the same root, demonstrating that this constraint was too weak. Therefore, we strengthened it in DQGen 2014 to ensure that the choices are not only distinct but dissimilar, that is, have different roots. Allowing more than one choice with the same root would make sense only for questions that focus on minor differences between them, for example to test knowledge of inflectional morphology.

3.1.2 Familiar

Distractors must be familiar to children; otherwise they may test vocabulary rather than reading comprehension (Cassels and Johnstone Reference Cassels and Johnstone1984). Educators use various metrics to quantify the difficulty of text and individual words. A common way to express these metrics is grade level, i.e., the grade at which students are expected to understand the text or know the word meaning. DQGen 2012 satisfied this constraint for ungrammatical and nonsensical distractors by generating them from the paragraph and a grade-leveled word list (Biemiller, Reference Biemiller2009), respectively. However, neither of these sources is large enough to provide enough candidate plausible distractors, because they cover too few of the Google N-grams used to test local coherence. Therefore, to exclude words likely to be unfamiliar to children, DQGen 2012 filtered out candidates whose unigram frequency falls below 5,000,000. We tuned this threshold by informal trial and error; higher thresholds proved too stringent to yield any plausible distractors.

In addition, DQGen 2012 constrained all four choices to be single words rather than phrases. Therefore, in using Biemiller's table to generate candidate nonsensical distractors, it filtered out the multi-word entries, such as ‘barbeque sauce.’

The pilot study and subsequent testing exposed shortcomings of previous strategies. In particular, the fact that a word has a frequency of at least 5,000,000 does not guarantee its familiarity to children. To bolster enforcement of this constraint, DQGen 2014 additionally requires all four choices to occur in the WordNet (Fellbaum Reference Fellbaum2012) database of 155,287 nouns, verbs, adjectives, and adverbs.

Moreover, one judge in the pilot study considered some words too difficult for children, such as ‘gauge.’ In fact, Biemiller (Reference Biemiller2009) rates the noun sense of this word at grade 2, but its verb sense (to estimate) at grade 10. These examples illustrate a limitation of DQGen 2012's methods to pick familiar words as distractors. It picked ungrammatical distractors from the words in the paragraph, nonsensical distractors from Biemiller's word list, and plausible distractors from Google N-grams, filtered by unigram frequency to avoid rare words. In all three cases, DQGen 2014 constrained words rather than word senses.

A more sophisticated approach would determine a distractor's word sense, or at least POS, when used to complete the sentence, and categorize the familiarity of its specific sense or POS. Tagging the distractor POS is easier than determining its word sense(s) when inserted in the sentence. Rating the familiarity of different word senses would require either a grade-leveled list of them like Biemiller's (2009), or a resource with information about the frequency of different word senses, such as the ordering of synsets in WordNet. Either way we’d need reliable POS tagging and word sense disambiguation to identify the distractor's POS and sense when inserted in the stem.

3.1.3 Meaningful

DQGen 2014 added lexical constraints to exclude stop words, homographs, and proper nouns to ensure that the answer and distractors are meaningful, that is, carry unambiguous semantic content and test comprehension.

Some words carry too little meaning to use as choices in cloze questions. In particular, stop words, modal verbs such as can, cannot, and will, and common verbs such as any form of be, do, have, and get, lack sufficiently specific semantic content to decide their relevance to the context. A related reason to avoid them as distractors, as a judge in the pilot study commented, is lest children notice that they are seldom the correct answer, and therefore eliminate them without considering them. DQGen 2014 implements these constraints by excluding the modal verbs are, be, can, cannot, could, did, do, does, get, got, gotten, had, has, have, is, may, might, must, shall, should, was, were, will, and would, as well as a list of common verbs and stop words (too many to list them all here).

Similarly, DQGen 2014 rejects homographs – words with the same spelling but different meanings and sometimes pronunciations – because their meaning is ambiguous. For instance, address can be a noun (location of a building) or verb (speak to someone). Thus, whether it is grammatical or makes sense depends on how it is read, rendering it unsuitable as a choice in a cloze question. To implement this constraint, DQGen 2014 excludes the homographs abuse, address, appropriate, august, bow, close, combat, combine, conduct, content, convert, does, dove, herb, lead, live, lives, minute, moderate, pate, pervert, present, presents, produce, project, read, record, sewer, sow, subject, tear(s), use, uses, wind, and winds.

Conversely, a proper noun such as the name of a person, location, date, or time, is so specific that including it as a choice in a cloze question is liable to test particular world knowledge rather than reading comprehension ability. To avoid this risk, DQGen 2014 excludes any word that starts with a capital letter.

3.2.1 Grammatical

As Table 2 shows, all three types of distractors involve grammaticality constraints. Ungrammatical distractors must make the completed sentence ungrammatical, e.g., ‘That helps your body find and kill are.’ In contrast, nonsensical and plausible distractors must make the completed sentence grammatical, e.g., ‘That helps your body find and kill terrorists.’ To check the grammaticality of a completed sentence, DQGen 2014 uses three natural language processing tools: the Stanford POS Tagger (Toutanova, Klein, Manning and Singer Reference Toutanova, Klein, Manning and Singer2003), the Stanford Parser (Klein and Manning Reference Klein and Manning2003), and the Link Grammar Parser (Sleator and Temperley Reference Sleator and Temperley1993).

DQGen 2012 used the Stanford POS Tagger (Toutanova et al. Reference Toutanova, Klein, Manning and Singer2003) to categorize an answer or distractor candidate as a noun, verb, adjective, or adverb. It required a plausible or nonsensical distractor to have the same POS as the answer, and an ungrammatical distractor to have a different POS. However, the pilot test revealed two problems with this scheme.

First, some words in English can have more than one POS. For example, the ungrammatical distractor ‘tutors’ in Figure 4 can be either a noun or a verb. DQGen 2014 therefore rejects an ungrammatical distractor if it has any POS that the answer can have, not just its tagged POS in the sentence. DQGen 2014 finds every possible syntactic category (e.g., noun, verb, etc.) of a candidate in WordNet; if the candidate is not in WordNet, it finds these categories in the British National Corpus.

Second, this categorization is too coarse to ensure grammaticality when desired, e.g., for nonsensical distractors. For instance, it fails to distinguish singular from plural nouns, and therefore misclassifies ‘in eight cauliflower’ as grammatical. To fix this problem, DQGen 2014 uses the Stanford POS Tagger to identify the correct answer's finer-grained POS, so it tags ‘cauliflower’ as a singular noun (NN), as opposed to a plural noun (NNS) like the correct answer ‘months.’

To be grammatical, a distractor must be appropriately inflected. However, all words listed in Biemiller's (Reference Biemiller2009) table are base forms. Therefore, a distractor generated from the table must be inflected into the appropriate form to match the POS of the correct answer, for instance, by converting a singular noun into plural form, or the base form of a verb into the appropriate number and tense. DQGen 2014 uses systematic transformation rules to inflect regular words, and table lookup to inflect irregular nouns and verbs.

Moreover, more than one POS can fit grammatically in the blank. For example, consider the sentence ‘Solar panels will collect the energy necessary for the electricity in the space _____.’ The answer ‘station’ is a noun, but the verb ‘left’ fits grammatically too. To test if a completed sentence is grammatical, DQGen 2014 therefore also uses the Link Grammar Parser (Sleator and Temperley Reference Sleator and Temperley1993), a syntactic dependency parser, as a heuristic grammaticality checker. It usually accepts grammatical sentences and rejects ungrammatical ones, especially for the short sentences typical of children's text. However, it sometimes fails to accept a grammatical sentence, as the last row of Table 3 illustrates.

Table 3.

Examples of grammaticality checking by Link Grammar Parser

As the ‘space station’/‘space left’ example illustrates, replacing an answer with a grammatical distractor may change the syntactic structure of the sentence, as Figure 5 shows. DQGen 2014 prohibits such changes lest they alter what the question actually tests. To enforce this prohibition, DQGen 2014 parses the completed sentence and compares its parse tree to the parse tree of the original sentence. If their syntactic structures differ, it rejects the candidate.

Fig. 5.

An example syntactic structure changed by replacing the answer with a distractor.

3.2.2 Normal

As a heuristic test of whether a completed sentence is plausible, DQGen 2012 checked whether its final 5-gram (including the end-of-sentence punctuation) occurs in the Google N-grams corpus (Brants and Franz 2006), implying that it is sufficiently normal English to appear at least forty times on the Web. For plausible distractors, the 5-gram consisting of the last four words of the sentence, followed by a period, must appear in this corpus. (For ungrammatical and nonsensical distractors, the last 4-gram of the completed sentence must not occur.) To enforce this constraint, DQGen 2012's source of candidate plausible distractors consisted of Google 5-grams of the form W X Y __. Here, W, X, and Y are the three words preceding the correct answer in the original sentence, e.g., ‘find and kill.’ If DQGen 2012 found fewer than five such 5-grams, it also allowed 4-grams of the form X Y __., e.g., ‘and kill __.’

In the pilot study, however, judges categorized only thirty-two per cent of the intended plausible distractors as plausible, and forty-two per cent as nonsensical. To reduce the incidence of such cases, DQGen 2014 checks if the final 5-gram preceding the period appears in the Google N-gram corpus, in which case the completed sentence is likelier to make sense. For instance, DQGen 2012 erroneously allowed ‘family’ as a plausible distractor for ‘We nip if they stray too far from ___.’ because the Google N-gram corpus includes the 5-gram ‘too far from family.’ In contrast, DQGen 2014 rejects ‘family’ as a plausible distractor because the corpus does not include the 5-gram ‘stray too far from family.’

The 5-gram constraint is often too strong, disqualifying all candidate plausible distractors. When this case occurs, DQGen 2014 relaxes the constraint to use just the final four words of the sentence, in an attempt to generate a cloze question anyway.

3.3.1 Relevant to stem

To score local relevance, DQGen 2012 measured the relatedness of a distractor to each word in the stem only by how often they co-occur in the same thirty-word window in British National Corpus. To aggregate this measure over the words in the stem, it used a Naïve Bayes formula:

$$\begin{equation*} \Pr (c| {stem) \propto \Pr (c)\prod \limits _{i = 1}^n {\Pr ({w_i}| c)} }. \end{equation*}$$

Here, c is a candidate plausible distractor and w_i is the ith content word in the stem. DQGen 2012 averaged these local coherence scores over the candidates, and allowed only candidates whose local coherence scores were above the mean, as a heuristic to increase the likelihood that they made sense locally.

This approach quantifies relevance as co-occurrence, thereby overlooking related words that seldom co-occur. To compute relevance more robustly, DQGen 2014 therefore instead averages the distributional similarity of a given word to the content words in the stem. DQGen 2014 uses DISCO (Kolb Reference Kolb2008, Reference Kolb2009) to calculate distributional similarity as the extent to which two words tend to co-occur in British National Corpus with the same other words, where co-occur means within three words of each other.

3.3.2 Irrelevant to preceding context

A good plausible distractor should not be too plausible. DQGen 2014 uses the same method to calculate relatedness to the context as to the stem, but constrains plausible distractors to be less relevant to the context than the answer is. DQGen 2014 ranks the candidates that satisfy this constraint by their global relevance (relatedness to the context), and chooses from the bottom half the candidate with the highest local relevance (relatedness to the stem). This heuristic gives local relevance priority over global (ir-)relevance; in comparison, DQGen 2012 chose the candidate with the lowest global relevance, which often turned out to be nonsensical because it was not locally coherent.

The purpose of DQGen 2014's heuristic is to pick a distractor that makes sense locally but not in the context of the preceding sentences, in order to detect failures of inter-sentential comprehension processes that monitor global consistency. To maximize the contrast in global relevance of the plausible distractor versus of the answer, how large a context should we consider? On the one hand, the longer the context, the more words related to the answer it may contain, driving up its relevance score. On the other hand, the further back in the text, the fewer such words are likely to occur. To quantify this tradeoff empirically, we computed the relevance of the answer to the context as a function of the number of sentences in the context. On average, answer relevance was greatest when the number of sentences was two. Therefore, DQGen 2014 first computes global relevance between a candidate and content words within a two-sentence context, and filters out any candidates whose global relevance is larger than the answer's global relevance. Next, DQGen 2014 ranks the rest of the candidates by their global relevance and considers the bottom half (the most unrelated to the context), from which it chooses the candidate with the highest local relevance (the most related to the stem). This heuristic maximizes the local relevance of plausible distractors subject to limiting their global relevance.

4.1.1 Descriptive statistics

We define the distractiveness of a distractor as the frequency with which students choose it over the correct answer. To analyze distractiveness, we computed the frequency of each response type within the 200 responses: correct answers (sixty-five per cent), plausible distractors (fourteen per cent), nonsensical distractors (eleven per cent), ungrammatical distractors (seven per cent), and non-responses (four per cent) where the child did not answer. This order is consistent with our expected order of distractiveness, namely plausible > nonsensical > ungrammatical.

Selecting a plausible distractor as correct: Plausible distractors are designed to make sense locally, so as to detect failure by the reader to relate the clozed sentence to the preceding context. For instance, consider this stem:

• • ‘In the Winter I do not hear many _______.’ [birds]

Some children chose ‘voices’ as the answer. This choice makes sense in the sentence, but is unrelated to the preceding context: ‘Some days the birds sing. In the Spring the birds get up early. They wake up long before I do.’

Selecting a nonsensical distractor as correct: The child who answered ‘lids’ to this question presumably did not understand the word, ignored the fact that lids can’t hear, or chose randomly.

Selecting an ungrammatical distractor as correct: Similarly, the child who answered ‘early’ didn’t know the word, ignored grammar, or chose randomly.

Null responses: In seven cases, the Reading Tutor presented the stem, but before it could present the choices, the student clicked Goodbye to exit in four cases, timed out in two cases, or clicked Back to return to the story menu in one case.

To examine further the relation of response type to distractiveness, we correlated the frequency of each response type against the child's reading level. For this analysis, N = 24 students, and frequency of a given type is the percentage of a student's responses of that type. E.g., the student who encountered the most questions (nineteen) had a reading level estimated by the Reading Tutor as second grade. This student had seven correct responses (thirty-seven per cent), three plausible responses (sixteen per cent), three nonsensical responses (sixteen per cent), four ungrammatical responses (twenty-one per cent), and two non-responses (eleven per cent). Correlation with reading level was negative for the percentage of ungrammatical responses (r = –0.41, p = 0.05), consistent with better readers being more sensitive to syntax (Hensler and Beck Reference Hensler, Beck, Ashley and Ikeda2006). Correlations with reading level were not significant for other response types, unsurprisingly given the limited sample size (N = 24 students).

4.1.2 Analysis of response types

To account for variance due to questions and students in testing for significant differences, we first used the glmer function in R to perform mixed-effects logistic regression with response type as a fixed effect, and stem and student as random effects. The dependent variable was whether the student chose that response type for that question. Like logistic regression, this type of model predicts a binary outcome – in this case, whether a distractor type children chose was affected – as the log odds ratio of the probability of choosing over the probability of non-choosing, but takes the variance of questions and students into consideration. Thus, the (admittedly non-independent) input observations for the regression consisted of N = 1,000 tuples, five for each response type – one tuple with an outcome of TRUE for the actual response, and four tuples with outcomes of FALSE for the other four response types. The random effects for student and stem turned out not to be statistically significant. This lack of statistical significance may imply that the results are likely to generalize to similar students and questions. On the other hand, it might just reflect low statistical power caused by insufficient data.

Dropping the non-significant effects eliminated both the random effects, yielding the logistic regression model (glm in R) whose coefficients appear in Table 4. As Table 4 shows, the base reference response Correct was significantly likelier than all four other response types. Table 4 does not show which if any differences among these four types are significant, but the differences in their beta coefficients suggest that responses are substantially likelier to be ungrammatical or null than plausible or nonsensical.

Table 4.

Logistic regression model of response selection; reference base for response_type is Correct

4.1.3 Analysis of response times

As a potentially more sensitive indicator of student performance, we also analyzed students’ response times to the different types of questions, which averaged about 30 sec overall. For analysis of response times, N = 193 non-null responses (to 116 distinct cloze questions), because response time is undefined for non-responses. To determine which differences were not only reliable but likely to generalize to unseen data from similar cloze stems and students, we used mixed-effects linear regression. To find the model that fit the data best, we used backward model selection, starting with the three available predictors that we expected could affect the outcome: response type as fixed effect, and stem and student as random effects (just their intercepts, not their slopes, which our data was too sparse to estimate). We kept removing the least significant predictor (the one with the highest p-value) until doing so stopped improving model fit in a Likelihood Ratio Test. Table 5 shows the resulting model.

Table 5.

Best-fitting model of response time

Random effect of individual student: Response time differed reliably by student (SD = 6.84), i.e., the best-fitting model had a distribution of random per-student intercepts (1 | student) ~ N(0, 6.84²).

No main effect of response: Response time differed only slightly, not significantly, by students’ responses:

• • Correct answer: M = 29.15 sec, N = 130, SD = 20.23 sec

• • Plausible distractor: M = 28.56 sec, N = 27, SD = 14.46 sec

• • nonsensical distractor: M = 33.50 sec, N = 22, SD = 25.93 sec

• • Ungrammatical distractor: M = 26.50 sec, N = 14, SD = 10.45 sec

No random effect of stem: We would have expected significantly different variance in response time among the 116 question stems. The absence of such an effect reassuringly suggests that the results are likely to generalize to future similar questions, but it might simply be an artifact of limited sample size.

4.2.1 Comparison methodology

To compare DQGen 2012 and DQGen 2014, we adapted the blind evaluation methodology used in the pilot study. This comparison differed from the pilot study in three ways.

First, all thirteen questions in the pilot study (including the question in Figure 1) came from a single story (2006). Due to its stronger constraints, DQGen 2014 selected only one question stem for ‘Tiny Invaders,’ so we needed additional questions in order to compare it to DQGen 2012. Besides, we had tuned DQGen 2014 to this text insofar as we had modified it to address problems identified in the pilot study. For a fair comparison, we needed to evaluate them on previously unseen texts. Out of the 282 Reading Tutor texts for which DQGen 2014 had generated at least one question, it had generated more than one question for twenty-six of them. We randomly chose six of these twenty-six texts to add. We had not previously run DQGen 2014 on these texts, so they had not affected its development, and hence constituted a fair basis for comparison. Appendix A.2 shows the sixteen stems that DQGen 2014 extracted from the seven texts (“Tiny Invader” plus six texts).

Second, the pilot study had presented only four choices to categorize for each stem. To compare the quality of the distractors generated by DQGen 2012 and DQGen 2014, we evaluated how often they were perceived as their intended types. This evaluation presented seven choices in random order, consisting of the correct answer, three distractors generated by DQGen 2012, and three distractors generated by DQGen 2014, without disclosing which was which. DQGen 2012 and DQGen 2014 generated the same distractor in only one case; for that question, we presented only six choices.

Third, the pilot study had also scored the overall quality of each question. We dropped this part of the evaluation because presenting seven choices for each stem precluded comparison of overall question quality between DQGen 2012 and DQGen 2014. However, as in the pilot study, we invited judges to comment on the candidate choices. There were thirteen such comments (two comments on correct answers, six comments on five distractors generated by DQGen 2012, and five comments on three distractors generated by DQGen 2014), some of which contributed to the error analysis in Section 4.4.

The Appendix shows our evaluation form. Five judges participated. All five were native English speakers and members of our research team. Two of them were also judges in the pilot study. The other three were unfamiliar with this study.

4.2.2 Result: judges’ agreement with each other and with intended distractor type

As in the pilot study, we used Fleiss’ Kappa to assess inter-rater agreement between categorizations by the five judges. Kappa was 0.63 for the version comparison experiment in Section 4.2, versus 0.58 for the pilot study in Section 2.2, and indicates ‘substantial agreement’ according to Landis and Koch (Reference Landis and Koch1977).

Figure 6 shows each individual judge's agreement with intended distractor type. Cohen's Kappa measures the agreement between two judges. We compared the individual ratings with the intended types of the distractors generated by DQGen 2012 and DQGen 2014 in order to understand how often each judge agreed with them. Figure 6 makes clear that all five judges agreed more often with DQGen 2014 (with Cohen's Kappa from 0.48 to 0.81) than with DQGen 2012 (with Cohen's Kappa from 0.22 to 0.44). Figure 3 showed earlier that in the pilot study, judges agreed with intended type more often for correct answers and ungrammatical distractors than for nonsensical and plausible distractors. Figure 7 showed that the same pattern held true in the study comparing DQGen 2012 and DQGen 2014. This study used different data than the pilot study – namely the subset of stems for which both versions of DQGen generated questions, and ratings from a different set of judges. Consequently, Figures 3 and 7 show different values for DQGen 2012.

Fig. 6.

Cohen's Kappa for each judge's agreement with intended distractor type in DQGen 2012 and DQGen 2014.

Fig. 7.

Cohen's Kappa for agreement with the intended choice type in DQGen 2012 and DQGen 2014.

We compared judges’ agreement with DQGen 2014 to their agreement with each other. As Table 6 shows, judges agreed slightly more often with DQGen 2014 (mean Cohen's Kappa of 0.65) than with each other (0.63).

Table 6.

Individual judges’ agreement with each other (Cohen's Kappa)

The confusion matrix in Table 7 shows the percentage of categorizations of distractors as ungrammatical, nonsensical, plausible, or correct for each intended type generated by DQGen 2012 and DQGen 2014. Table 7 has a row for each intended type, but no row for correct answers because they were the same for DQGen 2012 and DQGen 2014. Except for four categorizations as meaningful and one as nonsensical, the judges categorized every correct answer as Correct.

Table 7.

Confusion matrix comparing categorizations of DQGen 2012's and DQGen 2014's distractors

*Chi-square p = 0.001. †p = 0.004. ‡p = 0.34. Note: due to rounding, some rows do not sum to one hundred per cent.

Table 7 has a pair of columns for each category, showing the percentage of categorizations that agreed with the intended type in that row. The boldfaced percentages along the diagonal of the matrix are significantly larger for DQGen 2014 than for DQGen 2012. As the ‘<’ signs indicate, DQGen 2014 beat DQGen 2012 at generating distractors perceived as intended, for all three distractor types. Conversely, none of the off-diagonal percentages, representing disagreement with intended distractor type, are larger for DQGen 2014 than for DQGen 2012, except twenty-four per cent versus four per cent for the percentage of plausible distractors categorized as Correct. That is, all of the DQGen 2014 distractors intended to be plausible were grammatical and almost all of them made sense locally, but some of them were too plausible. The error analysis in Section 4.4 discusses the most frequent errors.

4.2.3 Generalizability of results

Our comparison of distractor quality necessarily used a limited number of judges (five), texts (seven), and question stems (sixteen). Can we be confident that our results generalize to similar unseen judges, texts, and stems? To find out, we used the glmer function in R to fit a logistic mixed-effects model. Like the logistic mixed-effects model in Section 4.1.2, this model has a binary outcome variable – in this case, whether a judge agreed with the intended type. The model predicts the log odds ratio of the probability of agreement over the probability of disagreement. Its fixed effects are version (DQGen 2012 or DQGen 2014), distractor type (ungrammatical, nonsensical, or plausible), and their interaction. The model includes individual judge and stem as random effects in order to test whether they account for significant variance in the results. It does not include text, because stem uniquely determines text. The sample for this analysis consisted of N = 2 versions × 3 distractor types × 16 stems × 5 judges = 480 categorizations (including five duplicate categorizations of the lone distractor generated by both DQGen 2012 and DQGen 2014). We used the same backward model selection procedure as in Section 4.1.3 above, which resulted in eliminating the interaction term and yielded the best-fitting model shown in Table 8. This procedure left stem and judge as random effects because eliminating them produced a worse model according to a likelihood ratio test. Controlling for these random effects means that they did not account for the significant fixed effects discussed below, but the fact that the random effects were significant means that the results might differ for similar stems or judges, i.e., drawn from the same distributions.

Table 8.

Best-fitting model of agreement; reference base for distractor_type is nonsensical

We now relate the model in Table 8 to the agreement rates in Table 7.

Main effect of version: DQGen 2014 significantly beat DQGen 2012 overall.

Main effect of intended distractor type: Compared to the quality of their nonsensical distractors, both DQGen 2012 and DQGen 2014 generated significantly better (p < 0.001) ungrammatical distractors, with a trend (p < 0.1) toward worse plausible distractors.

No interaction of version with distractor type: That is, including such an interaction does not predict performance better than the combined fixed effects of version and distractor type.

Random effects: Performance differed reliably by stem (SD = 0.40) and judge (SD = 0.39). That is, agreement varied systematically by individual stems and judges. Consequently, we cannot assume that our results would generalize to similar stems or judges. To achieve more generalizable results, future research could improve judge selection and instructions to reduce inter-rater disagreement, and identify features of stems that cause disagreement with intended distractor types.

4.3.1 Comparison methodology

To evaluate DQGen 2014 against human performance, we had to specify the task being performed and the criteria by which to evaluate it. Given a text with some sentences selected to turn into multiple choice cloze stems by deleting the last word, the task was to generate a distractor of each type – ungrammatical, nonsensical, and plausible. To enable controlled evaluation of distractors, we gave DQGen 2014 and humans the same seven texts and stems listed in Appendix A.2 and used in Section 4.2's comparison to DQGen 2012.

Our principal evaluation criterion was whether a generated distractor achieved its purpose according to human judges blind to its source (DQGen or human), its intended type (ungrammatical, nonsensical, or plausible), and the correct (original) answer. An additional evaluation criterion was time: we wanted to know how long it took humans to categorize or write each type of distractor. Besides quantifying the relative difficulty of categorizing versus writing the three types of distractors, the practical purpose of this information was to predict which would be faster – writing distractors by hand, or hand-vetting distractors generated by DQGen.

Apparatus: The pilot study and the comparison of DQGen 2014 to DQGen 2012 had presented questions in an Excel spreadsheet for judges to fill in their responses, but those studies had each used fewer than ten judges, and they did not capture timing information.

To overcome both limitations, we developed a website to run the experiment. We implemented the website in PHP and connected it to a MySQL database server that logged a timestamped event for each page entrance or exit, keyboard input, or menu selection. The database also kept track of each participant's position in the protocol in order to continue at the same point after an interruption, and to avoid repeating any of the protocol.

Participants: To recruit human judges proficient in English and sufficiently knowledgeable about reading comprehension to categorize and write distractors, we posted a request to Carnegie Mellon's doctoral Program in Interdisciplinary Educational Research (www.cmu.edu/pier) and to the Society for the Scientific Study of Reading (triplesr.org). The request directed participants to the website for the experiment.

After data cleaning to filter out data from in-house software testing, failed attempts to log in, unfinished protocols, and two null categorizations, we had data for twenty-seven participants’ completed experimental protocols.

Procedure: The experimental protocol consisted of logging into the experiment website, a brief introduction, the two main tasks (first categorizing, then writing), and finally a survey with a series of optional typed-input questions about various aspects of the experiment.

The introduction thanked participants for ‘helping our research by doing two tasks: rating (the first task) and designing (the second task) multiple choice cloze (fill-in-the-blank) items to assess children's reading comprehension.’ It explained that in the first task, they would read texts containing a total of eight cloze stems, see different candidate completions of each stem, and categorize each completion as correct, plausible, nonsensical, or ungrammatical. It showed the annotated example in Figure 1 and said:

In the categorization task, participants read 3–4 texts containing a total of eight cloze stems. Cloze stems appeared on a new screen with this note: ‘If you need to reread the text first, please click on the Previous button above. Otherwise, click on one of the four buttons below to classify the following completion (independently of the others).’ The button for each category included its description shown in Figure 1. Participants categorized seven candidate single-word completions, one at a time, for each cloze stem, e.g., ‘The next morning, Silly Pilly was ready to go to ____.’ The seven candidates, reordered randomly for each participant, consisted of the correct answer (school), the three distractors generated by DQGen 2014 (along, slang, and breakfast), and three authored by humans (e.g., blue, slip, and home). The writing task was similar:

To avoid problematic input such as null responses, typos, and non-words, we included code to reject them and prompt for a replacement, but these events, averaging 25 sec, occurred for only eleven of the 504 human-written distractors in our data.

Assignment to conditions: All participants did categorization before writing, which we considered harder and in fact averaged about three times as long per completion. We wanted to give judges some experience in thinking about different types of distractors before writing them. The resulting order bias was mitigated by not explicitly labeling distractors by source or intended type. Although judges might conceivably have imitated the eight human- and eight DQGen-generated distractors they rated for each intended type, the fact that they classified so many of them as other than their intended types suggests that any such imitation was limited.

To avoid text-specific bias, we counter-balanced the study design so that half the participants (the ‘AB’ group) categorized completions for set A (stems 1–8) and then wrote completions for set B (stems 9–16), and the other half (the ‘BA’ group) categorized completions for the stems in set B and then wrote completions for the stems in set A.

Participants in each group categorized the same distractors generated by DQGen 2014, but they categorized different distractors authored by humans, so as to give us a more diverse sample. To limit the protocol duration, each participant categorized distractors authored by only one participant from the other group. Accordingly, we used the following algorithm to assign participants to categorize human-authored distractors.

The first participants saw distractors written by staff experienced with cloze questions. However, as soon as participants completed the protocol, the distractors written by these protocol completers became available for subsequent participants to categorize. Once a participant completed the protocol by writing distractors for set B, another participant was assigned to categorize them, and to write distractors for set A. Similarly, those distractors were eventually (if ever) categorized by some subsequent participant assigned to categorize set A and write distractors for set B, and so on.

This ‘daisy-chaining’ algorithm assigned each new participant to categorize cloze choices from whichever set (A or B) had been categorized so far by fewer participants who had finished the protocol. It chose human-authored distractors not yet categorized by anyone who had finished the protocol. Consequently, all twenty-seven participants categorized distractors generated by DQGen 2014 for either set A or set B. Twenty-one participants’ distractors got categorized – sixteen participants with one judge per distractor, and the other five with two. Our data set contains no categorizations for the remaining six participants’ distractors, either because nobody categorized them, or because we discarded data from other participants who may have categorized them but didn’t finish the rest of the protocol.

4.3.2 Results: agreement with intended distractor type

Like Table 7 in Section 4.2.2, Table 9 shows the percentage of categorizations of each intended distractor type as Ungrammatical, nonsensical, Plausible, or Correct, except that it compares DQGen 2014 to humans instead of to DQGen 2012. Table 9 is based on 1,486 categorizations by twenty-seven judges of sixteen correct answers, forty-eight distractors generated by DQGen 2014, and 504 distractors written by twenty-one humans. Inter-rater reliability was substantial on DQGen 2014's distractors (Fleiss’ Kappa = 0.66). Only forty human-authored distractors were categorized by more than one judge, namely five sets of eight distractors categorized by a pair of judges. Cohen's Kappa for each pair of judges averaged 0.46 (N = 5, SD 0.25), i.e., only moderate agreement, versus 0.60 (SD 0.09), close to substantial agreement, on the eight DQGen-generated distractors they both categorized, but the two means did not differ reliably on a paired T-test (p = 0.31).

Table 9.

Confusion matrix for categorizations of DQGen 2014's and human distractors and correct answers

*Chi-square p < 0.001. †p = 0.089. ‡p = 0.053.

The boldfaced diagonal entries in Table 9 compare the percentages of categorizations that agreed with the intended types of DQGen 2014- and human-generated distractors. To determine which differences were not only reliable but likely to generalize to unseen data from similar cloze stems and judges, we used a logistic mixed-effects model, just as in Section 4.2.3. To find the model that fit the data best, we used backward model selection, starting with five predictors we expected could affect the outcome. Three were fixed effects: distractor source, intended type, and their interaction. Two were random effects: stem and judge. We kept removing the least significant predictor (the one with the highest p-value) until doing so stopped improving model fit in a Likelihood Ratio Test. We now relate the resulting model in Table 10 to the agreement rates in Table 9.

Table 10.

Best-fitting model of agreement; reference base for distractor_type is nonsensical

Main effect of intended distractor type: Compared to their nonsensical distractors, both DQGen 2014 and humans generated significantly worse (p < 0.02) plausible distractors, with a trend (p < 0.1) toward better ungrammatical distractors.

No main effect of source: Surprisingly, DQGen 2014's distractor quality did not differ significantly overall from humans.’

Interaction of source with distractor type: Although DQGen 2014 and humans did not differ significantly overall, they differed for some distractor types after adjusting for the fixed effect of distractor type and the random effect of stem. DQGen 2014's ungrammatical distractors were significantly (p < 0.001) better than humans’, its plausible distractors were probably (p ~ 0.05) worse than humans’, and there was a trend (p < 0.1) for its nonsensical distractors to be better than humans’.

No random effect of individual judge: We would have expected a judge effect if some judges were systematically worse, e.g., categorized at random. The absence of such an effect reassuringly suggests the results are likely to generalize to future similar judges – perhaps because the comparison to human used twenty-seven judges, versus the five judges used in the comparison to DQGen 2012.

Random effect of stem: Performance differed reliably by stem (SD = 0.35), i.e., the best-fitting model had a (1 | stem) ~ N(0, 0.35²) distribution of random per-stem intercepts. For some stems, judges could not tell correct answers from plausible distractors, as the error analysis in Section 4.4 below will discuss further.

4.3.3 Time analysis

We analyzed the time for humans to write and categorize distractors, both to measure the difficulty of the task, and the practicality of using DQGen 2014 to assist rather than replace human authors.

To see whether DQGen could speed up human authoring, we compared the time for humans to categorize versus write distractors. They averaged about 5 sec to categorize a choice and about 19 sec to write any type of distractor. Based on Table 1, vetting a distractor generated by DQGen and rewriting it only if unacceptable would average (5 sec) + (1 – agreement) × (19 sec) = about 10 sec, barely half of the 19 sec to write it by hand. Moreover, ninety-two per cent of distractors would match their intended type if vetting is perfect, i.e., categorizes DQGen-generated distractors properly by definition. Only seventy-three per cent of human-authored distractors do so.

To analyze the time to categorize a choice, we used mixed-effects linear regression starting with source, intended type, agreement, and their interactions as fixed effects, and stem and judge as random effects. Backward model selection led to the model in Table 11 and summarized below.

Table 11.

Best-fitting model of time to categorize a choice

No main effects: Categorization time didn’t differ significantly by source, intended type, or agreement.

Random effects: Categorization time differed significantly by stem (SD = 1.75) and judge (SD = 1.36).

Interaction of agreement with intended type (p < 0.001): Categorization was significantly faster when it agreed with choices intended to be correct (4.6 sec < 8.7 sec), ungrammatical (4.3 sec < 7.2 sec), or nonsensical (5.2 sec < 5.7 sec). For plausible distractors, categorization was slower (6.6 sec > 5.8 sec) (albeit not significantly) when it agreed with intended type, perhaps because confirming that a distractor is plausible requires additional thought.

4.4.1 nonsensical distractors

nonsensical distractors were miscategorized most often (seventeen per cent) as Ungrammatical. The nonsensical distractor miscategorized as Ungrammatical the most often (eighteen of its nineteen categorizations) was ‘share’ in:

• • ‘We nip if they stray too far from _______.’ [home]

This example illustrates an issue that arises when a distractor can have more than one POS. DQGen 2014 chooses nonsensical distractors to have the same POS as the correct answer, in this case the noun ‘home.’ The word ‘share’ is often a verb, but can also be used as a noun. Perhaps judges categorized it as Ungrammatical because they considered only its verb POS. Another possibility is that they thought of its noun sense, but considered it a countable noun and hence ungrammatical unless preceded by ‘a’ or ‘the.’

More generally, the boundary between syntax and semantics may be blurry and vary by individual. To illustrate, consider the nonsensical distractor ‘slang’ generated by DQGen 2014 for this sentence:

• • ‘the next morning, Silly Pilly was ready to go to _______.’ [school]

The word ‘slang’ is a noun, but the verb ‘go to’ also imposes semantic constraints: the grammatical object of ‘go to’ should be a location or event (at least when used literally rather than metaphorically, as in ‘go to pot’). This constraint may explain why seven of nineteen judges categorized ‘slang’ as Ungrammatical.

The point illustrated by this example has a methodological implication for similar future studies: namely, instructions to judges may need to define ‘ungrammatical’ more precisely. It also has a technological implication: generating distractors to distinguish among sensitivity to surface-level syntax (e.g., POS and number agreement), semantic dependency relations (e.g., constraints on semantic role fillers, such as verb-object compatibility), and deeper types of coherence (e.g., based on inference from a situation model) would require further natural language processing such as semantic role labeling or inferring dependency relations.

Although nonsensical distractors were miscategorized most often as Ungrammatical, they were miscategorized six per cent of the time as Plausible. The nonsensical distractor miscategorized the most often (nine of its eighteen categorizations) as Plausible was ‘pressure’ in:

• • ‘And now, each in our way, and with God's help, we must answer the _______.’ [call]

Half the judges categorized this metaphorical use of ‘pressure’ as Plausible. Why did DQGen 2014 consider it nonsensical? It assumes that every meaningful sequence of five words occurs often enough (at least forty times) in Google's trillion-word corpus to appear in Google N-grams, so a sequence that does not must be nonsensical. This example illustrates the fallibility of this heuristic assumption: even the 3-gram ‘answer the pressure’ is not in Google N-grams. Realizing that ‘answer the pressure’ is meaningful would require deeper natural language understanding, especially of metaphorical usage.

4.4.2 Plausible distractors

Plausible distractors were miscategorized twenty-one per cent of the time as nonsensical (i.e., not plausible enough) and twenty-two per cent of the time as Correct (i.e., too plausible).

The plausible distractor miscategorized most often as Correct (seventeen of its nineteen categorizations) was ‘democracy’ in this line from Bill Clinton's First Inaugural Address, Wednesday, January twenty first, 1993:

• • ‘our people have always mustered the determination to construct from these crises the pillars of our _______.’ [history]

The actual correct answer was ‘history,’ but ‘democracy’ really fits just as well. This example illustrates a drawback of defining the correct answer as the word used in the original sentence: this heuristic occasionally leads to a trick question, i.e., one with a distractor just as valid as the original word. DQGen 2014 assumes that the correct answer fits better than topically unrelated distractors. This assumption fails when the topicality constraint fails to disqualify a plausible distractor.

Similarly, sixteen and fifteen, respectively, of nineteen judges miscategorized ‘globe’ as Correct in two sentences earlier in the same speech:

• • ‘Now, the sights and sounds of this ceremony are broadcast instantaneously to billions around the _______.’ [world]

• • ‘We earn our livelihood in peaceful competition with people all across the _______.’ [earth]

As two judges commented, ‘globe’ fits in both cases. In both examples, the distractor is a synonym for the correct answer. An obvious improvement is to disqualify synonyms for the correct answer as plausible distractors.

Two plausible distractors were tied for being miscategorized most often as nonsensical (twelve of nineteen categorizations):

• • ‘ride’ in ‘Corky's had a long ride and she needs a _______.’[run]

• • ‘computer’ in ‘It took six great big strong guys to load it all into the _______.’ [truck]

In both cases the final 5-gram preceding the period, but with the distractor filled in, occurs in Google N-grams, and the distractor is topically related to the stem. In the first case, ‘ride’ sounds awkward because it also occurs earlier in the same sentence, and seems nonsensical because a dog who just had a long ride wouldn’t need one. In the second case, ‘computer’ seems nonsensical because loading something into a computer doesn’t require physical strength. Both distractors are nonsensical because they contradict world knowledge, so detecting them as such would presumably take deeper interpretation.

4.4.3 Human-authored distractors

This paper focuses on automated generation of good distractors, but the human-authored distractors provide some useful insight because there were several of each type for each stem. As in Section 4.4.1 and 4.4.2, we examine the cases with the most miscategorizations, except that they were distributed over multiple human-authored distractors of the same type, not just one distractor.

There were six cases of ungrammatical human-authored distractors miscategorized as Plausible in this sentence:

• • ‘Now, the sights and sounds of this ceremony are broadcast instantaneously to billions around the _______.’

Six of thirteen judges classified ‘brave,’ ‘flows,’ ‘light,’ ‘politics,’ or ‘run’ as nonsensical here. Evidently, the judges parsed them as nouns, but their authors did not (except politics). DQGen 2014 considers alternative parts of speech, which may help explain why it outperformed humans in generating ungrammatical distractors.

There were also six cases of human-authored plausible distractors miscategorized as nonsensical in this sentence:

• • ‘Their miserable shack was transformed into a luxurious _______.’

Six of thirteen judges categorized ‘car,’ ‘cushion,’ ‘hovel,’ and ‘train’ as nonsensical here. All of these distractors are countable concrete nouns and hence legitimately plausible in the local context, yet the judges considered them nonsensical.

Conversely, there were six cases of human-authored nonsensical distractors miscategorized as Plausible in:

• • ‘We nip if they stray too far from _______.’

Those distractors were ‘England,’ ‘heaven, ‘rivers’, ‘sheep,’ and ‘water,’ each of which makes sense out of context. Thus, here it was the authors who were apparently unable to ignore the preceding context in deciding if distractors are nonsensical.

DQGen 2014 by its very design disregards the context when generating nonsensical or ungrammatical distractors. In contrast, writing or judging nonsensical distractors is evidently a difficult task for humans who know the context, because they have trouble disregarding it. Another possibility is that they simply ignored or forgot the qualifier ‘meaningful by itself’ in the first place, and therefore did not even try to disregard the preceding context.

One implication is that depriving humans of the context would make both tasks easier for them. Another implication is that some of the sixteen distractors generated by DQGen 2014 to be plausible but classified by humans as nonsensical in sixty-two of 196 categorizations might in fact be plausible out of context. Thus, depriving humans of context might not only make it easier for them to judge plausibility, but increase their accuracy as well.