Participants

The research was conducted at a private language school in Auckland, New Zealand. A total of 12 intact classes with 12 different teachers and 118 learners participated in the study. All 12 teachers were native speakers of English. They had between 1 and 15 years of teaching experience and held a variety of English-as-a-second-language qualifications, such as the Certificate in English Language Teaching for Adults and Master of Arts in Applied Linguistics. Eight of the teachers were male and four were female. The only demographic information gathered directly from the learners was their nationality, with the majority (over 75%) coming from Korea, China, Japan, and Taiwan. Females accounted for 56% of the participants and males accounted for 44%. The classes ranged in attendance from 3 to 14 learners. The English language proficiency of the learners, as measured by an in-house placement test, ranged from low to upper intermediate.

Procedure

A total of four lessons were observed over a 1-week period for each of the 12 classes. The researcher was present during all observations as a nonparticipant observer, and the observations were audio-recorded using a wireless cassette recorder with a clip-on microphone attached to the teacher. This arrangement recorded all teacher-learner interaction, whether as a whole class, in small groups, or one-on-one; however, it did not capture learner-learner interaction when the teacher was not present. Because it was important to identify individual learners' utterances on the recordings, detailed notes consisting of learners' names and speech excerpts were made during the observations and matched to the recordings. A total of 32 hours (1,917 minutes) of meaning-focused classroom interaction were observed and recorded. However, in an effort to minimize the disruption of the testing procedures, only the data gathered from the second and third observations were used for the individualized testing, although the first and fourth observations were used to supplement the description of the occurrence of incidental focus on form in these classes. Thus, a subset of 17 hours (1,022 minutes) of data was used for testing. (The testing procedures will be described in detail in the following section.)

Although instruction at this language school consisted of both meaning-focused and form-focused lessons, the present study focused exclusively on meaning-focused activities, which were defined as those activities having a primary goal of exchanging information, rather than learning about or practicing specific linguistic forms (Pica, Kanagy, & Falodun, 1993). This definition, however, was not provided to the teachers and no effort was made to guide them in their choice of lesson plans beyond asking them to observe meaning-focused lessons. Activities observed included information- and opinion-gap tasks, role plays, and class discussions on a variety of topics. Because the study focused only on the four observed lessons, it is not possible to comment on the nature or content of lessons that were not observed, and it is acknowledged that the observed lessons comprised only a small part of the learners' instruction.

Identification and Testing of Focus on Form Episodes

Following the observations, the FFEs were identified. A FFE was defined as consisting of “the discourse from the point where the attention to linguistic form starts to the point where it ends, due to a change in topic back to message or sometimes another focus on form” (Ellis et al., 2001a, p. 294). Thus, in practice, a FFE started either when a learner produced a linguistic error that was addressed by the teacher or when a learner queried a linguistic item.

An example of a FFE is shown in (2). The learner and teacher are discussing which of four prisoners should be paroled early (an information- and opinion-gap task). The FFE (which is italicized; see the Appendix for a full transcription key) begins when the learner makes an error in the verb form, which is immediately corrected by the teacher. The learner repeats the correct form, and they continue discussing the prisoner's case. Thus, the participants' attention has switched briefly from meaning-exchange to a linguistic item and then returned to the main message. To determine reliability of FFE identification, a second rater coded a sample of 10.3% of the data (three lessons totaling 199 minutes), with a resulting agreement rate of 89.9%.

Because it was not possible to predict which linguistic items would be targeted in a particular lesson, the FFEs had to serve as a type of pretest, demonstrating—either through an error or a query—that a learner had difficulty with a particular linguistic structure. Once the FFEs were identified, individualized test items relating to the linguistic items targeted in the FFEs were constructed, and the learners who were responsible for triggering the specific FFEs were tested on those items. Tests were given on two different occasions: one immediate (1 to 3 days after the FFE) and one delayed (13 to 15 days after the FFE). However, because a pilot study suggested that the tests served to focus attention on the linguistic forms, it was not possible to conduct immediate and delayed tests on the same linguistic items. Therefore, immediate testing was done on linguistic items from the second observation and delayed testing on items from the third observation. Thus, for each class, only linguistic items from two of the four observations were used for testing. Additionally, the immediate testing (based on the second observation) was conducted after the third observation (which provided the data for the delayed test) so that the testing would not heighten learners' awareness of subsequent FFEs and thus affect their responses to the delayed testing. Finally, note that the linguistic content of instruction between the observations and the delayed testing was not documented; although the incidental and extensive nature of the linguistic structures targeted in the FFEs suggests that the majority of structures would not have been attended to, it is possible that some structures might have been focused on in subsequent lessons.

For the administration of the tests, learners were withdrawn individually from class into a quiet room where the researcher presented the test items to them. The test items were based as closely as possible on the FFEs. However, to facilitate testing, three templates with slightly differing formats—suppliance, correction, and pronunciation—were developed. For suppliance and correction tests, the researcher read aloud the test items to the learners. To replicate as closely as possible the oral nature of the FFEs, learners were not given written copies of the test items. Each test item was read to the learner no more than twice; however, learners could respond after the initial presentation if they wished. For pronunciation tests, students were asked to read aloud the test items, because of the difficulty of eliciting the targeted items without providing written prompts. The learners responded orally to all test items, and each testing session was audio-recorded.

Suppliance.

Suppliance tests were used primarily for FFEs targeting vocabulary and spelling and required learners to provide linguistic information. For example, the test rubric in (3), which the researcher read aloud to the learner, asks the learners to supply the meaning of increasing. The test item was based on episode 456, presented in (4), in which the learner asked about the meaning of this word during a discussion activity.

Suppliance tests could also ask learners to provide a word or phrase that corresponded to a particular meaning or definition, such as the test rubric in (5), in which the learner needed to supply the response addicted. To reduce the chance of learners providing an acceptable answer other than the target of the FFE, learners were told the first letter of the target word (Laufer & Nation, 1999). The FFE (episode 462) on which this test item was based is shown in (6).

Correction.

In correction tests, learners were asked to improve sentences that they had produced incorrectly during FFEs. For example, in the test rubric in (7), a learner was given the opportunity to correct his preposition error, which was the focus of the FFE, episode 648, shown in (8).

Pronunciation.

Pronunciation tests were the only tests that involved written prompts. Learners were first asked to read aloud a sentence containing the targeted word or phrase and then to read aloud the word or phrase in isolation. Although the written form of the word provided learners with clues about pronunciation, it was decided that this was an unavoidable limitation because of the difficulty of eliciting the targeted items without some type of prompt. Thus, the learner's mispronunciation of scalp in episode 400, shown in (10), is tested by the test rubric in (9) that asks him to read aloud first his original utterance and then the word in isolation.

Because learners were tested only on the FFEs in which they participated directly (either by making an error or raising a query), the number of test items for each learner varied. Some learners had no test items on a particular day, whereas the highest number of test items was 14. (See Loewen, 2003, for a further description of the variation in learner participation in incidental focus on form.)

Scoring of Test Items

Learners' responses to the test items were scored as correct, incorrect, partially correct, other correct, assisted correct, or inconclusive.

Correct.

The learner produced a response that correctly matched the targeted linguistic item in the FFE. For example, in response to the test item in (11), the learner provides the correct form, depressed, which was based on episode 612, shown in (12).

Incorrect.

The learner does not correctly produce the linguistic item targeted in the FFE. The example in (13) illustrates an incorrect response, alcoholer, as well as the response that would be expected, alcoholic. This example is based on episode 618, which is presented in (14).

Partially Correct.

The learner produces a response that improves on the targeted linguistic error in some way but is still not entirely accurate. This rating was also given to pronunciation test items in which one context (e.g., the sentence) was incorrect, but the other context (e.g., isolated) was correct. For example, in (15) [based on episode 585, which is presented in (16)], the learner mispronounces nephews in the sentence but pronounces it correctly in isolation. Thus, the test item is scored as partially correct.

Other Correct.

The learner corrects the targeted error, but in a way that differs from the linguistic information provided in the FFE. Thus, in the example in (17), the learner has provided the arguably acceptable word attached instead of assigned, which was provided in episode 138 [shown in (18)].

Test Reliability and Validity

With any type of language testing, it is important to establish the reliability and validity of the tests (Douglas, 2001). However, with this type of individualized testing, it was not possible to use traditional means of establishing reliability, such as test-retest or internal consistency. Consequently, an alternative approach to establishing reliability was taken: Potential sources of error variance or measurement error were identified (Brown, 1996) and an attempt was made to neutralize them. (See Loewen, 2002, for a complete description of the efforts to ensure reliability.) However, the tests still comprised different linguistic information and different numbers of items. Thus, it was decided that the best way to determine the suitability of the test items was to judge their construct validity, particularly given that reliability is often seen as a component of validity (Douglas).

Construct validity refers to “the extent to which we can interpret a given test score as an indicator of the ability(ies) or construct(s) we want to measure” (Bachman & Palmer, 1996). The aim of the present study was to investigate to what extent the test items measured learners' ability to produce the linguistic information they were provided in FFEs. Thus, one way to establish construct validity was to determine how well the test items related to what occurred in the FFE. Although the specific linguistic items targeted in the FFEs were varied and unpredictable, the FFEs, nevertheless, had certain characteristics in common. Thus, the validity of the test templates—suppliance, correction, pronunciation—was investigated in order to determine if they could be used satisfactorily on a variety of linguistic items. With that purpose in mind, a panel of three experts in the field of SLA and L2 testing was convened. The panel members were provided with transcripts of a random sample of approximately 10% of the tested FFEs. The panel members then listened to the audio-recorded test items and were asked to rate on a 4-point scale (1 corresponded to highly appropriate, whereas 4 corresponded to inappropriate) how well the items tested the linguistic information targeted in the FFEs. The results from this rating session showed that the average rating for the suppliance and pronunciation test items was highly appropriate (mean = 1.17) and that there was considerable agreement (94.7%) among the three raters. However, the correction test items did not have similarly high ratings or agreement. It was therefore decided to conduct a second rating session in which all of the correction test items (n = 149), plus several other test items that concerned the researcher (n = 19), were rated by the same three raters. A total of 21 test items had means over 2.5, which was considered to indicate low construct validity; these test items were withdrawn from the analysis. Reasons given by the panel for low validity scores included the decontextualized nature of the test items in relation to the highly contextualized nature of the FFE interaction as well as the inconclusiveness of some FFEs in which negotiation of linguistic forms was never fully resolved. Finally, the reliability of the coding of the test responses as correct, incorrect, and so forth was confirmed by a second rater, who scored a random sample of 21% of the test items. This resulted in a reliability level of κ = .850, which is above the .80 level suggested by Davies et al. (1999).

Coding of FFEs

After the FFEs were identified and tested, they were transcribed in detail and coded for a variety of characteristics, based primarily on the analysis in Ellis et al. (1999). Table 1 identifies and defines these characteristics.

Characteristics of FFEs

An example shown in Table 2 will help illustrate the coding system. In episode 560, presented in (19), the participants are discussing which of four candidates should receive a heart transplant (an information- and opinion-gap activity). The learner makes a grammatical error in the formation of the plural childrens, which the teacher recasts. The learner incorporates this feedback successfully into his production; however, the teacher provides another piece of linguistic information by informing the learner of what is wrong with his utterance (no S). Thus, the episode is complex and heavy because the teacher provides two different response moves. The FFE is code-related because there is no apparent breakdown in communication.

Example of coding scheme

To investigate the reliability of the coding, a second rater coded a sample of 21% of the FFEs (n = 277). Table 3 lists the kappa reliability and percentage agreement scores for each of the characteristics. The kappa coefficients ranged from κ = .672 to .915, and all percentage agreement scores were over 85%.

Reliability scores

Statistics

Raw frequencies and percentages were calculated for the test response categories. Because the data consisted of frequency counts of categorical data, Pearson's chi-square analysis was performed on the raw frequencies. An alpha level of p < .05 was set for all chi-squares. Because analyses involved variables with more than two coding categories (i.e., the chi-square was larger than 2 × 2), adjusted standardized residuals of greater than plus or minus 2.0 were used to identify significant differences. All inferential statistics were performed using SPSS 10.0.

To investigate causality in the data, a binary logistic regression analysis was performed (Hatch & Lazaraton, 1991; Saito, 1999). Logistic regression is similar to linear regression; however, it is distinguished by the fact that the dependent variable is binary or dichotomous rather than continuous (Greenhouse, Bromberg, & Fromm, 1995). The purpose of logistic regression is to find the “best fitting and most parsimonious” model to describe the relationship between a dependent variable and independent (or predictor) variables (Hosmer & Lemeshow, 2000). In the case of the present data, it was desirable to use regression modeling to determine which characteristics of incidental focus on form would best predict the test scores. Because the dependent variables were categorical rather than continuous, logistic regression was appropriate. Finally, because of the slightly different structures of the suppliance, correction, and pronunciation tests, it was decided to conduct separate regressions for each template.

Given that logistic regression allows only binary dependent variables, it was necessary to dichotomize the dependent variable (in this case, the test scores). Because both partially correct and other correct responses contained an element of correctness, it was decided to recode them as correct. This recoding increased the sample size and an exploratory analysis with and without such recoding produced nearly identical results. Furthermore, the assigning of binary values to the independent variables makes the interpretation of the resulting odds ratios (which will be discussed later in this section) easier. Table 4 details the coding scheme for the dependent and independent variables and indicates the value assigned to each coding category.

Binary coding scheme

The method of entry for each variable in the regression equation was forward stepwise, which involves adding the independent variables to the model one at a time in an attempt to produce the best and most succinct model. Because each step adds the variable that makes the greatest change to the model, stepwise regression is particularly appropriate when the relationships between the predictor variables and the dependent variable have not been previously investigated or are relatively unknown. If a variable does not make a significant contribution to the model, it is not included (Hosmer & Lemeshow, 2000). Thus, a forward stepwise regression produces a model that accounts for the largest proportion of the dependent variable using the fewest—but most important—independent variables. Hosmer and Lemeshow argued that setting an alpha level of .05 for a forward stepwise regression is too stringent and might exclude important variables; instead, they “highly recommend” choosing an alpha level in the range of .15 to .20 (p. 118). For the purposes of the current study, an alpha level of .15 was chosen.

The output of a logistic regression analysis includes odds ratios (ORs) and 95% confidence intervals for each independent variable. ORs are important because of their relationship to the regression coefficient (OR = e^β1) and their ease of interpretation. “The odds ratio then approximates how much more likely (or unlikely) it is for the outcome to be present among those with x = 1 than among those with x = 0” (Hosmer & Lemeshow, 2000, p. 50). For example, if the dependent variable of test score (coded as 0 = incorrect and 1 = correct) and the independent variable of complexity (coded as 0 = simple and 1 = complex) have an OR of 2, then the OR estimates that correct test scores are twice as likely to occur when based on complex FFEs than on simple ones in the dataset. Conversely, an OR of 0.5 estimates that correct test scores are one-half as likely to occur when based on complex FFEs than on simple FFEs in the dataset.

Finally, it should be mentioned that one assumption of chi-square and logistic regression analyses is that the data are independent, meaning that “each observed frequency is generated by a different subject” (Gravetter & Wallnau, 1991). This assumption, often violated in SLA research, can lead to Type I errors (i.e., the overestimation of significant differences in the data; Hatch & Lazaraton, 1991; Saito, 1999). In the present study, the test scores are not entirely independent because multiple scores are produced by individual learners (i.e., individual learners often had more than one test item). To test for the effect of this arguably limited dependence of data, a participant variable for learners was created and included in the logistic regression models, as suggested by Saito. The results indicated that although the learner variable was a significant predictor of test scores in the correction and pronunciation tests, the ORs and significance levels for the other independent variables entered into the model did not change substantially when the participant variable was excluded. These results suggest that although the data are not completely independent, any dependence in the data does not substantially affect the regression models. (See Loewen, 2002, for further discussion of this issue.)