Item Consistency

As evinced in examples (1) and (2), lexical matching creates a high level of internal consistency within an item. The target or critical region in the sentence (i.e., the verb study/studies in [1]) is as similar as possible in each version. Additionally, the words following the verb (. . . in the) are also the same. This is necessary given that the processing of a critical region in a sentence oftentimes continues or “spills over” onto the words immediately following the critical region. This is known as the spillover effect (e.g., Rayner & Duffy, Reference Rayner and Duffy1986), and the regions that follow the critical region are called spillover regions. In some sentence processing studies, including those conducted on monolinguals, the effect of a tested variable first appears in a spillover region, not in the target region. Thus, the design of the spillover region is as important as the design of the critical region. Likewise, regions prior to the critical region should be as similar as possible across conditions given that any difficulties inherent in processing precritical regions could cause unintended spillover effects in critical regions. For example, in a study of English subject-verb agreement conducted with English monolinguals, Wagers, Lau, and Phillips (2009, Experiment 1) found that adverbs that immediately followed subject nouns took longer to read when the subject was plural (e.g., The old keys unsurprisingly were . . .) versus when it was singular (*The old key unsurprisingly were . . .). Without the intervening adverb, the spillover processing would have occurred on the critical verb, which would have made the results difficult to interpret. These findings suggest that all versions of item (2) need an intervening word or phrase between the subject and verb.

Just as it is important to create consistency within an item, it is equally important to maintain consistency across all experimental items in a study, especially with respect to the critical and spillover regions. For example, the critical verbs to be included in the hypothetical subject-verb agreement study should appear in the same position in every item and, to the extent possible, should be similar in length (i.e., number of characters or syllables) and frequency. The simple reason for this is that longer words and less frequent words take longer to read than shorter words and more frequent words.^{Footnote 2} Likewise, the nouns that serve as subjects of the target verbs should also be similar in length and frequency. Additionally, the words that make up the spillover regions should be as similar as possible, if not identical, across items. That is, in the subject-verb agreement study, it would be ideal for the critical region to be in the in every item. Where this is not possible, alternate words should be of the same grammatical category and identical in length (e.g., at the, on the, by the). An additional point to consider when testing L2 learners is knowledge of the target words. In our view, far too few L2 processing studies provide independent assessments of learners’ knowledge of the critical words used in experimental items. Using only high-frequency target words—as determined by a corpus or frequency dictionary—reduces the chance of including words that are unknown to participants but is not a substitute for independently verifying word knowledge, especially when testing lower proficiency learners.

Sentence Norming Studies

For most anomaly-detection studies, following the guidelines discussed so far will yield a methodologically sound set of experimental items that will elicit the anticipated effects in a control group without the need for pilot testing. This is because most (morpho)syntactic anomalies are categorically unacceptable and easy to create. In contrast, when the experimental manipulation involves a lexical, semantic, pragmatic, or plausibility bias, it may be useful to conduct a sentence norming study to ensure that the experimental items work as intended. In a norming study, participants recruited from the same population as the control group—but who do not participate in the main experiment—are asked to rate or judge sentences on a psychometric scale, usually a three-, five-, or seven-level Likert-type scale.^{Footnote 3} Depending on the purpose of the norming study, the sentences may be identical to those intended for use in the main study or may contain keywords or sentence frames that will be used to create the experimental items for the main study. The numerical scores associated with participants’ ratings are averaged, and the means are used to determine whether the sentences (or parts thereof) are suitable for inclusion in the main study. If not, revisions are made, and a second norming study with a new group of participants from the same population is conducted to test the revised items.

Roberts and Felser (Reference Roberts and Felser2011) used a norming procedure to confirm the reliability of the plausibility manipulation used in their experiment. To summarize, their study investigated the influence of plausibility information on the resolution of temporary subject-object ambiguities (i.e., garden paths) in English. Their stimuli consisted of paired items such as those in (3). In the (a) versions, the critical noun phrase was a plausible direct object for the preceding verb, and in the (b) versions, it was not (compare song in [3a] to beer in [3b]).

1. (3) a. While the band played the song pleased all the customers.

2. b. While the band played the beer pleased all the customers.

To ensure that the (b) versions of items were considered less plausible than the (a) versions, Roberts and Felser administered an offline plausibility rating questionnaire in which a subgroup of native English controls rated the plausibility of sentences such as The band played the song and The band played the beer on a scale from 1 “very plausible” to 7 “very implausible.” The analyses confirmed a reliable difference between the two conditions. Additional examples of norming procedures used in L2 processing research can be found in Havik, Roberts, van Hout, Schreuder, and Haverkort (Reference Havik, Roberts, van Hout, Schreuder and Haverkort2009) and Siyanova-Chanturia, Conklin, and Schmitt (Reference Siyanova-Chanturia, Conklin and Schmitt2011).

Number of Items

The quantity of items to include in a study is an important point to consider. Regardless of online method, some data will be lost due to participant error or distraction. Furthermore, as is discussed in a later section of this article, some data must be removed before conducting statistical analyses. Researchers using behavioral methods such as self-paced reading and eye-tracking typically lose 8–15% of collected data, and significantly more data are lost in ERP studies (see Morgan-Short & Tanner, Reference Morgan-Short, Tanner, Jegerski and VanPatten2014). Additionally, participants in sentence processing studies never read more than one version of the same item. That is, a participant in the subject-verb agreement study described previously would read either version (1a) or version (1b) of item (1), but not both. Reading both versions of the same item would cause repetition effects, which refer to when participants respond to a stimulus in an unnatural way (e.g., reading it superficially or deliberately slowly) because they have seen it before. To minimize such effects, researchers divide the different versions of the experimental items into separate presentation lists. A participant only reads sentences from one of the possible lists. The number of lists equals the number of versions in the items; that is, a study that uses experimental doublets like (1) requires two presentation lists, and one that uses experimental quadruplets like (2) requires four presentation lists. Each list must contain an equal number of sentences from each condition. The practice of rotating different versions of items across different presentation lists and then rotating the lists across participants is called counterbalancing. The following list presents the division of items for a study that tests the experimental doublets that appear in (1). The (a) versions are grammatical and the (b) versions are ungrammatical.

• List 1: 1a, 2b, 3a, 4b, 5a, 6b, etc.

• List 2: 1b, 2a, 3b, 4a, 5b, 6a, etc.

The division of items for a study that tests the experimental quadruplets in (2) is as follows (with versions [a] and [c] being grammatical and versions [b] and [d] being ungrammatical):

• List 1: 1a, 2b, 3c, 4d, 5a, 6b, 7c, 8d, etc.

• List 2: 1b, 2c, 3d, 4a, 5b, 6c, 7d, 8a, etc.

• List 3: 1c, 2d, 3a, 4b, 5c, 6d, 7a, 8b, etc.

• List 4: 1d, 2a, 3b, 4c, 5d, 6a, 7b, 8c, etc.

Psycholinguists usually create enough items for participants to read 8–12 items per condition, the goal being to have data for at least 6–10 items per condition after accounting for data loss. Therefore, studies that test items with two conditions require 16–24 items. Studies testing items that appear in four conditions require 32–48 items. In contrast to self-paced reading and eye-tracking, ERP studies require 30–40 items per condition due to the larger amounts of data loss associated with this method (Morgan-Short & Tanner, Reference Morgan-Short, Tanner, Jegerski and VanPatten2014). Therefore, an ERP study that tests experimental doublets requires 60–80 items, and one that tests experimental quadruplets requires 120–160 items. Complete sets of experimental items used in sentence processing research do not always appear in published articles, due to space limitations; however, it is helpful to reviewers to include them in articles submitted for peer review. For examples of complete sets of experimental doublets that follow the design considerations discussed so far, see the appendices in Frenck-Mestre and Pynte (Reference Frenck-Mestre and Pynte1997), Williams (Reference Williams2006), and Roberts and Felser (Reference Roberts and Felser2011), among others. Complete sets of experimental quadruplets can be found in the appendices of Cunnings and Felser (Reference Cunnings and Felser2013), Dussias and Piñar (Reference Dussias and Piñar2010), and Jackson and Dussias (Reference Jackson and Dussias2009), among others.^{Footnote 4}

Describing Item Creation

Even when items are lexically matched in appropriate ways, researchers in our field sometimes describe them in ways that suggest otherwise. For example, when discussing the creation of experimental doublets, authors often provide a description and example such as the following: “Our study tested 24 grammatical sentences such as (4) and 24 ungrammatical sentences such as (5) for a total of 48 test sentences.

1. (4) The student studies in the library on campus.

2. (5) *The student study in the library on campus.”

The consecutive numbering of the examples suggests that the two sentences belong to separate items when, in fact, they are different versions of the same item. This particular study tests 24 items (not 48), each of which appears in two versions: (a) and (b). In sum, to avoid confusion, stimuli should be described as consisting of some number of items (not sentences), each of which appears in some number of versions or conditions. The example should show one item in all of its conditions (a, b, c, d), as shown previously in (1) and (2).

Distractor Task Types

Two of the most common types of poststimulus distractor questions are acceptability judgments and meaning-based comprehension questions, though other types of binary-decision tasks have been employed as well. Judgment tasks have enjoyed long-standing popularity in psycholinguistics, in part because they yield additional offline data that is interesting to most researchers, but this practice has recently been questioned by some SLA scholars. Their concern arises in part from the assumption that metalinguistic tasks such as grammaticality or acceptability judgments can increase the application of explicit knowledge during the primary experimental task of reading. It is difficult to argue against this point of view, particularly when an acceptability judgment is used as the distractor task for an experiment that tests L2 learners’ sensitivity to violations of grammatical rules that are taught and tested in language classes, as is the case for most structures that are the focus of anomaly-detection experiments. Indeed, it has been shown empirically with self-paced reading that the online processing behavior of L2 learners can vary according to whether the distractor task is an acceptability judgment or a meaning-based comprehension question. In one instance, Leeser, Brandl, and Weissglass (Reference Leeser, Brandl, Weissglass, Trofimovich and McDonough2011) found that intermediate-level L2 learners of Spanish exhibited online sensitivity to violations of noun-adjective gender agreement when the secondary task was a grammaticality judgment but not when it was a meaning-based comprehension question. The findings of the Leeser et al. study also speak to the importance of the instructions that participants receive in advance of completing an online task. If one’s aim is to test for implicit sensitivity to a particular type of violation, participants should not be told or led to believe that they will read erroneous sentences.

Another concern with acceptability judgments is regarding the validity of the task itself as an activity that falls outside the realm of daily language comprehension in the real world. In other words, real-time measurement of the behavior and brain activity of L2 learners while they are determining the acceptability of a sentence is probably not as widely generalizable as is the real-time measurement of the behavior of L2 learners who are engaged in meaning-driven language comprehension. Still, there is evidence that the secondary task does not always affect experimental outcomes in self-paced reading so dramatically. For instance, Jackson and Dussias (Reference Jackson and Dussias2009) and, later, Jackson and Bobb (Reference Jackson and Bobb2009) studied the processing of complex wh-questions with case marking in L2 German. The first study used a grammaticality judgment as a distractor task and the second used meaningful comprehension questions, but both studies found a nativelike initial preference for object extraction among L2 participants. Despite the overall similarity between the outcomes of the two studies, however, only the L2 participants in the investigation that used the grammaticality judgment probe exhibited nativelike reading time effects later on in the sentence while reading a second, subordinate clause. It is also worth noting that all of the critical stimuli in both of these studies were grammatical, unlike the agreement stimuli for the Leeser et al. (Reference Leeser, Brandl, Weissglass, Trofimovich and McDonough2011) study described previously, so the type of distractor task may be less critical with sentence processing paradigms that do not involve errors or other easily recognizable phenomena. It also appears that the waveforms observed in ERP studies, typically obtained when sentences are processed in conjunction with an acceptability judgment task, can also be observed when the only task is to passively read for comprehension and there is no acceptability judgment (Osterhout, Allen, McLaughlin, & Inoue, Reference Osterhout, Allen, McLaughlin and Inoue2002). Thus, there is the potential for the distractor task to affect experimental outcomes, and these effects may vary further according to research method and the linguistic structure tested. With self-paced reading in particular, the evidence suggests that acceptability judgments are inappropriate as a distractor task, and eye-tracking would probably be similar in this regard, but with ERPs, the available evidence regarding the potential role of the distractor task is much more limited. The choice of poststimulus task should therefore be intentional, so that the task is consistent with the objectives of the investigation and with prior work using the same research method.

Frequency of Distractors

Another consideration regarding the distractor task is whether the distractor probes appear after all of the stimuli for an experiment or after just a certain percentage of stimuli chosen at random. Both formats are common, and the choice to include poststimulus questions or decisions in some or all experimental trials depends on the purpose of the distractor task in a given experiment. If the only reason for including the secondary task is to keep participants engaged in reading the stimuli, then it is probably not necessary for questions or decisions to appear with every single trial, and limiting the number of poststimulus queries to one for every three or four stimuli, for instance, can save quite a bit of time (if that is a concern). In contrast, the inclusion of distractor questions or decisions consistently across all trials in an experiment can have some important advantages, including improved distraction from the primary experimental measure and research objectives, better face validity (because most linguistically naïve participants would accept answering questions as a language-related task but probably do not see the reading of sentences as a language measure in and of itself), and more consistent attention paid to stimuli throughout the experiment. Last, and most important, accuracy and reaction time data from the poststimulus task items can be additionally informative with regard to sentence processing behavior, but only if sufficient data are available. Thus, if this is an objective, then there should be distractor task questions or decisions after every stimulus.

Of course, the advantages of having a poststimulus distractor task can be inadvertently undermined if the task items are not appropriate for the experiment at hand, so the creation of individual questions or decisions for the task is very important. Every effort should be made to create and pilot poststimulus queries that do not introduce confounding effects or extraneous variability in the data. Specifically, distractor task items should be in the same language as the stimuli to avoid the increased crosslinguistic influence that may arise in bilingual or mixed-language contexts (Grosjean, Reference Grosjean2008). Additionally, it is most typical to have just one type of distractor task in a given experiment because the inclusion of more than one type of task (e.g., some stimuli followed by a plausibility decision and others followed by an argument-interpretation question) could introduce task-switching costs as a confounding variable. Furthermore, the content of the poststimulus items should also be neutral with regard to the linguistic variables that are manipulated in the stimuli. For instance, in two self-paced reading experiments, Havik et al. (Reference Havik, Roberts, van Hout, Schreuder and Haverkort2009) found that the processing preferences of L2 learners of Dutch reading temporary subject-object ambiguities like those in (6) were influenced by the type or frequency of comprehension questions that followed the stimuli.

1. (6) a. Subject Relative: Short

2. Daar is de machinist die de conducteurs heeft bevrijd uit het brandende treinstel.

3. “That is the engine-driver who the guards has saved from the burning train carriage.”

4. b. Object Relative: Short

5. Daar is de machinist die de conducteurs hebben bevrijd uit het brandende treinstel.

6. “That is the engine-driver who the guards have saved from the burning train carriage.”

7. (7) Poststimulus Comprehension Question (True-False Semantic Verification)

8. De machinist bevrijdde de conducteurs.

9. “The engine-driver saved the conductors.”

Specifically, these researchers observed a nativelike subject relative preference among the L2 participants—who also would be expected to show the same preference in their native German—only when each of the experimental stimuli was followed by a true/false semantic verification like that in (7), and even then only among those L2 participants with a higher working memory span and only with short relative clauses. When a second experiment was run with the same materials but with only 25% of stimuli followed by a semantic verification and with only 25% of those items specifically targeted at the interpretation of subjects and objects, the L2 participants did not show any online processing preference for the subject or object relative clause. Another notable difference between the two experiments was that the first contained 64 experimental stimuli and only 16 distractors, whereas the second included 64 critical items and 64 distractors. It is interesting to note that the native Dutch participants in the study did not appear to be affected by the type and frequency of poststimulus distractor questions, as their reading behavior was consistent across both experiments, which suggests that these aspects of experimental methodology may be more crucial in L2 research than in mainstream psycholinguistics.

Another way in which extraneous effects might inadvertently affect experimental data is if distractor task items repeat the target form from the stimuli that precede them or if they draw participants’ attention to the semantic content of the target form. To illustrate, if an experiment targets verbal agreement with an error recognition paradigm, as exemplified in (8) (repeated from [1] for convenience), and if the poststimulus query repeats the target form without the error, as the examples in (9) do, this may draw additional attention to the error present in the stimulus, it may provide an external reminder of the correct form, or it can otherwise affect participant behavior while reading subsequent stimuli. The question in (10) is similarly inappropriate, in this case because answering it correctly requires participants to focus on the semantic content of the verb, thus encouraging participants to selectively pay more attention to the verbs in sentences. This is especially true when the same type of question appears after stimuli throughout the experiment, as it encourages participants to develop a task-specific reading strategy. Another problem with comprehension questions that repeat part or all of the stimulus verbatim is that they are not a good measure of sentence comprehension because it is possible to respond correctly without processing word meaning and because they target the meaning of only one part of the sentence. The comprehension questions in (11), however, use synonyms and paraphrasing to go beyond the surface form of the stimulus to test higher level processing (while still using language that is no more complex than that of the stimuli). This type of question avoids pitfalls associated with repeating or targeting a specific linguistic form in the stimulus and also is a better gauge of meaningful comprehension. (Note that the question format, such as yes/no, true/false, or other binary-choice options, is not relevant here because these and other formats are all acceptable.)

1. (8) Agreement Stimulus from (1): Grammatical and Ungrammatical Conditions

2. a. The student studies in the library on campus.

3. b. *The student study in the library on campus.

4. (9) Comprehension Questions: Inappropriate, Target Form Repetition

5. a. Does the student study in the library?

6. Yes No

7. b. This person studies in the library at school.

8. True False

9. (10) Comprehension Questions: Inappropriate, Focus on Verb Semantics

10. a. Does the student sleep in the library?

11. Yes No

12. (11) Comprehension Questions: Appropriate

13. a. This library is at a school, probably at a university.

14. True False

15. b. Does the event described take place in a restaurant?

16. Yes No

A couple of final methodological points regarding the poststimulus distractor task are the timing of responses to these items and the provision of feedback on the accuracy of responses. Although many published papers do not indicate whether distractor questions were timed, it is common to limit the amount of time that a participant can take to answer a question or make a decision to 10,000 or 15,000 ms or even less, depending on how much additional reading is required. This can be done in one of two ways: Either the experiment can be programmed such that the question screen times out and the experiment continues after a set time limit, such as 10,000 ms, or the participants can take unlimited time to respond and the data can be trimmed later on. Building a time limit into the experiment is probably preferable, as it discourages excessive contemplation or reflection during the experiment, and it improves the efficiency of research sessions. Some L1 researchers also opt to provide feedback when responses to poststimulus queries are inaccurate (e.g., Gibson & Wu, Reference Gibson and Wu2013), presumably to maximize participants’ level of attention while reading stimuli, but it is not clear yet what effect feedback may have with L2 learners.

Self-Paced Reading

In selecting which self-paced reading data to analyze and report, one important decision for the researcher is whether or not to include reading time data from trials with incorrect distractor task responses. It has, thus far, been fairly standard practice in sentence processing research to eliminate such data, under the assumption that inaccurate responses to basic comprehension questions (i.e., ones that all normal adults would be expected to respond to accurately) and similarly transparent queries reflect either a lack of attention during the experiment or comprehension behavior that is otherwise nonstandard. There are necessary exceptions to this assumption, however, when there is no expectation for consistently accurate responses, perhaps because of task difficulty level or because the type of distractor task is such that there is no objectively correct response (e.g., when the task is an ambiguity interpretation), so this type of data cleaning is not always appropriate. Additionally, when the data corresponding to incorrect responses are substantial in number, as is more common with L2 learners, these can be reserved for independent analyses (see, e.g., Juffs & Harrington, Reference Juffs and Harrington1996), or, when the data from incorrect responses are not as numerous, statistical analyses can be run once on data from both correct and incorrect trials and once without data from the incorrect trials, to see if there are any differences. Although such exploratory analyses are not as common in sentence processing research with normal adult populations of native speakers (but see, among others, Gibson, Desmet, Grodner, Watson, & Ko, Reference Gibson, Desmet, Grodner, Watson and Ko2005; Hsiao & Gibson, Reference Hsiao and Gibson2003) and thus far have not been very common with research on L2 sentence processing either, they are of particular interest with any population that may regularly exhibit comprehension difficulty, as is the case with L2 learners, and thus should not be discounted a priori.

The next step in preparing the reading time data for statistical analyses with traditional ANOVAs is trimming to minimize the effects of outliers or extreme data points. (If residual reading times are to be calculated, as described later in this section, that calculation can come before or after trimming, depending on the trimming method used. With mixed-effects models, described at the end of this section, data trimming is not as important and is therefore minimal to nonexistent.) There is no single acceptable technique for data trimming, but it usually involves the deletion of reading times of less than 100–200 ms, which are quite rare, as well as the replacement of very high values with more moderate ones. Outlying high values can be designated in one of three ways: (a) via an absolute cutoff in the range of 2,000–6,000 ms (depending on the length of the stimulus region), (b) with a variable cutoff that is 2–3 standard deviations above the mean reading time for each group or individual participant in each stimulus region and in each condition, or (c) with a combination of both methods. Where intersubject variability is high, as is often the case with nonnative processing, the identification of outliers by individual participant and item—rather than by group—will lead to greater experimental power (i.e., will reduce the chances of avoiding a Type II error; Ratcliff, Reference Ratcliff1993) but takes more time. Once the outliers have been identified, the values are then either replaced by a more moderate number that is typically the same as the cutoff or removed entirely, leaving missing values. One potential concern with data trimming is that it can result in too many missing values for a given participant or item in any of the stimulus conditions (i.e., less than six remaining reading times on which to base an aggregate mean, described in the next paragraph), particularly when data points from trials with inaccurate distractor task responses have already been deleted, so it is usually preferable to replace outliers rather than delete them.

Once the reading time data have been selected and trimmed, they can be averaged and submitted to statistical analyses. In both cases, each region of interest from the stimuli is treated as a distinct measure, with separate descriptive and inferential statistics. One notable point of difference between psycholinguistic reading time studies of L2 acquisition and most other types of SLA research is the addition of items analysis to the traditional subjects analysis, which entails double the amount of averaging and analyzing. To begin the analysis by subject, aggregate means are calculated for the critical stimulus region, once for each participant for each stimulus condition. Each aggregate mean is computed by averaging the reading times for all the stimuli read in a given condition by the participant, so a simple two-condition experiment like that illustrated in (1) would have two aggregate means per participant (one for all of the grammatical items read and another for all of the ungrammatical items read), whereas a four-condition experiment like that in (2) would have four means per participant. Descriptive statistics normally include grand means for each group in each stimulus condition, which are averages of all the individual means from the participants in each group. To complete the subjects analysis, the processes of calculating aggregate means and then group grand means are repeated for each stimulus region that is of potential interest, which often includes a precritical region in which no effects are expected; the critical region, in which the initial processing effect is predicted to occur; the spillover region; the sentence-final region; and any intervening regions. The spillover region is of potential interest because reading time effects can carry over from one stimulus region to the next, whereas the last region of a sentence is the site of additional processing known as sentence wrap-up (Just & Carpenter, Reference Just and Carpenter1980). In addition to causing generally longer reading times, sentence wrap-up can also be the site of delayed or echoic reading time effects caused by the experimental manipulation in the stimuli. For the items analysis, the whole process is repeated for each stimulus region of interest, but, instead of calculating aggregate means for each participant, means are calculated for each stimulus item for each participant group in each stimulus condition. Thus, an experiment with two stimulus conditions and three participant groups would have six aggregate means per stimulus item.

Statistical analyses of reading time data are traditionally ANOVAs and t tests, which are conducted separately for each stimulus region of interest, once by subject and once by item. Analyses by subjects are distinguished from analyses by items by the use of subscripted numerals. F ₁ and t ₁ refer to analyses conducted by subjects, and F ₂ and t ₂ refer to analyses conducted by items. The purpose of such statistical tests is to determine whether there are real differences in reading times among stimulus conditions and whether these effects vary among participant groups. Thus, a typical set of ANOVAs by subject will each have repeated measures and will include the one or two linguistic variables manipulated in the stimuli as within-subjects variables (e.g., from example [2], grammaticality: grammatical and ungrammatical; subject number: singular and plural) plus the participant grouping variable as a between-subjects variable (e.g., group: L1, L2 advanced, and L2 near-native). For the corresponding ANOVAs by item, the variables and levels of variables would remain the same, but they would all be repeated-measures or within-subjects (“within-items”) variables because the same item contributes to all levels of all variables. Generally speaking, a main effect is deemed significant when F and t tests are significant by subjects and by items (a variation of the technique proposed by Clark, Reference Clark1973), but it is common in sentence processing studies to obtain a significant effect in one of the two analyses and a marginally significant or insignificant effect in the other. With mixed-effects models, subjects and items are both incorporated into a single set of tests, so the interpretation of statistical significance is more straightforward because there is only one p value to consider.

In some studies, it may not be possible for the critical or spillover region to be equal in length across the different versions of the experimental items. This is true of the subject-verb agreement experiments depicted in (1) and (2), in which the critical verb varies in length from five to seven characters (study vs. studies). Differences in length introduce confounds that could make the results difficult to interpret. For example, reading times on the ungrammatical verb study in (2b) may not be longer than reading times on the grammatical verb studies in (2a), as expected if readers display online sensitivity to such errors, because the ungrammatical version is shorter than its grammatical counterpart. Similarly, participants may spend more time reading the ungrammatical verb studies in (2d) relative to the grammatical verb study in (2c) because it is longer, not because they have detected a violation of person-number agreement. Additionally, participants (including monolinguals) read at different rates, which introduces interparticipant variability to the computation of group means. One way to control for differences in length across otherwise comparable conditions and individual differences in reading rates is to calculate residual reading times, also called deviations from regressions (Ferreira & Clifton, Reference Ferreira and Clifton1986; Trueswell, Tanenhaus, & Garnsey, Reference Trueswell, Tanenhaus and Garnsey1994). Residual reading times are calculated separately for each participant and sentence region by submitting the raw reading times from each participant for all segments of all items (i.e., critical and noncritical) to a regression equation with word length as the predictor variable. Reading times predicted by the participant’s regression equation are then subtracted from the raw reading times obtained in the experiment proper. Positive numbers refer to reading times that were slower than expected, and negative values refer to reading times that were faster than expected. The resulting residual reading times are then trimmed and statistically analyzed using the same procedures as for the raw times.^{Footnote 6} So, when it is necessary to adjust for length or individual reading rates, researchers should conduct both sets of analyses. In contrast, individual reading speed is known to correlate with some other individual differences, such as L2 proficiency, working memory, lexical access, and semantic integration (Hopp, Reference Hopp2013), so residual reading times may be inappropriate for the investigation of individual differences. In studies in which both raw and residual times are analyzed and the patterns of significance are identical or highly similar, or in which manuscript space is a concern, researchers often report one set of statistical analyses (i.e., either the raw or the residual reading times), noting any differences in the other set as appropriate (for an example of this practice, see Chen, Gibson, & Wolf, Reference Chen, Gibson and Wolf2005).

Finally, the data from the offline distractor task are also of interest and therefore analyzed and reported. Aggregate means for global accuracy, with data from both critical stimuli and fillers included in the scores, are reported and compared statistically by subject and by item using t tests or ANOVAs, usually to show the overall range of comprehension among the L2 participants and how their performance compared to that of native speakers. Where there are group differences, these should be taken into consideration later on, when making a final interpretation of the overall outcome of the experiment. Additionally, similar sets of statistical tests are performed on the reaction times and accuracy scores from only those distractor questions that followed the experimental stimuli. Experimental power can be improved with these tests by trimming the response time data from the distractor questions with the same method used to trim the reading time data from the stimuli, as already discussed. Analysis of poststimulus distractor question data is potentially informative because the question is essentially a spillover region for the experimental stimulus, so there may be delayed processing effects that surface in the response times or accuracy. For instance, meaning-based comprehension questions can sometimes be read more slowly when they follow ungrammatical sentences than when they follow grammatical sentences, and accuracy rates for meaning-based comprehension questions can also be slightly but significantly lower with ungrammatical sentences than with grammatical sentences (e.g., Jegerski, Reference Jegerski2012). As with the primary reading time data from the experimental stimuli, these effects can also vary among participant groups, which would be evident in statistical interactions.

Although L2 psycholinguistics has, thus far, relied primarily on the type of ANOVAs just described for statistical hypothesis testing, recent developments in the field of statistics have catalyzed a fairly rapid shift in mainstream psycholinguistics toward mixed-effects modeling as the standard method of analysis, and there is good reason to think that a similar shift is underway in L2 processing research. There are several important advantages to mixed-effects modeling that make it especially appealing for experimental research on human language. Perhaps most important, it provides a considerable improvement on previous methods for treating experimental items as well as participants as random effects (the importance of which was articulated by Clark, Reference Clark1973), especially with independent variables that are continuous as opposed to categorical. Additionally, as compared to traditional parametric testing, mixed-effects modeling is less affected by missing data points and outliers and can be used to analyze both normal and binomial data distributions (with linear and logit models, respectively). Finally, from the practical standpoint that is the focus of this article, mixed-effects modeling is also more efficient because it requires fewer steps and less time to prepare raw data for analysis. There is no need for data trimming or for the calculation of aggregate means by participant and by item. Data output files are simply compiled into a master sheet and imported into an appropriate statistical software program, such as R (R Core Team, 2013), in which any necessary sorting to remove trials with incorrect comprehension question responses or transformations can be carried out prior to mixed-effects analysis (described in greater detail by Cunnings, Reference Cunnings2012).

Eye-Tracking

Analysis of eye-tracking data involves the same steps required of self-paced reading, albeit with a considerably larger number of dependent variables. In contrast to self-paced reading, which provides one online reading time measure for each region of interest, eye-tracking provides two broad categories of data: (1) reading/fixation times (reported in ms), which can be fractionated into various subcomponents that include early and later measures of processing, and (2) regressive (i.e., backward) eye movements out of or into a region of interest, which are typically reported as the proportion of the sentences in an experimental condition in which such movements occurred out of or into the critical region (for details about the various eye-movement measures, see Keating, Reference Keating, Jegerski and VanPatten2014). Eye-tracking studies usually report one or two of the so-called early measures and one or two later measures. Thus, for each region of interest, researchers analyze several dependent variables, the choice of which is determined by the goals of the study and the point in the eye-movement record at which predicted effects are likely to show up.

Other than the differences in the total number and types of dependent variables available for analysis, self-paced reading and eye-tracking data are handled in similar ways. First, when analyzing fixation times and regressions out of or into a region, it is standard practice to remove from analyses all trials for which the poststimulus distractor question was answered incorrectly (barring the same exceptions mentioned previously for self-paced reading). It is also common practice to screen fixation time data for each measure (separately by subject and by item) to identify outlying values. Outliers can be identified by establishing cutoff values or using the standard deviation method. Fixations shorter than 50–100 ms are usually removed or merged with a neighboring fixation under the assumption that such short fixations are not indicative of any meaningful cognitive behavior. Establishing an absolute cutoff for an outlying high value depends on the type of measure (early vs. later) and other factors such as length of the region and so forth (see Keating, Reference Keating, Jegerski and VanPatten2014). In addition to using absolute cutoffs to remove extreme values, researchers may also use the standard deviation method (calculated separately by subject and by item) to remove values in a given condition that exceed some number of standard deviations above or below the participant or item mean (usually 2 or 2.5 SDs) for each condition in the study. Once identified, outlying fixation times are either replaced with an alternate value (e.g., the cutoff value, when using an absolute cutoff procedure, or the participant or item mean for the condition, when using the standard deviation method) or removed entirely and left as missing values. Once outliers are treated, the data for each dependent variable are analyzed using ANOVAs and t tests, which are conducted separately for each region of interest, once by subjects and once by items, as explained previously for self-paced reading. The practice of deriving and analyzing length-corrected residual reading times is not common in eye-tracking studies. When this technique is employed, it is usually only conducted on gaze durations or first-pass reading times. As with self-paced reading, mixed-effects modeling is fast becoming the standard method of analyzing eye-tracking data in studies of monolingual sentence processing (e.g., Cunnings, Patterson, & Felser, Reference Cunnings, Patterson and Felser2014). Finally, data from the offline distractor task are analyzed both for global comprehension accuracy and as a locus for potential spillover processing from the experimental sentences, as described previously for self-paced reading.

ERPs

The first step toward summary and analysis of ERP data from a sentence processing experiment is to select the subset of data that is of interest. The researcher must decide both how many electrodes to record data from during the experiment, typically 19–64 (Morgan-Short & Tanner, Reference Morgan-Short, Tanner, Jegerski and VanPatten2014), and which of those electrodes to include when reporting the research. It is common to report data from only those electrodes predicted to be most informative, which usually number around nine, although data from as few as three electrodes can be sufficient in shorter articles (e.g., Tanner, Osterhout, & Herschensohn, Reference Tanner, Osterhout, Herschensohn, Chandlee, Franchini, Lord and Rheiner2009). The investigator must also decide whether to include data from all trials, regardless of response accuracy, or to exclude data from those trials for which the participant provided an incorrect response to the distractor question. The inclusive method is more common in L2 research.

The next step is to filter and clean the electroencephalography (EEG) data to generate the ERPs for each electrode. Raw EEG recordings are, by nature, very noisy because each electrode measures total activity from an area of the cerebral cortex that contains numerous neurons, each a potential source of electricity. Even something as minute and unconscious as blinking or making saccadic eye movements during a linguistic experiment entails brain activity that generates extraneous voltage fluctuations in the data. Thus, prior to presenting summarized data or conducting statistical analyses, the ERP—the electrical activity associated with a particular cognitive event of interest—is extracted through a series of steps, the first of which is cleaning and filtering the EEG recordings. For the amplified data from each electrode, frequency filters are applied and the specific time interval of interest surrounding each experimental stimulus for each participant is separated out into what is known as an epoch. Epochs containing artifacts, or extraneous waveforms, especially those associated with any type of muscle movement (e.g., blinking), are excluded from further analysis. Next, the EEGs are averaged across all trials of the same type (i.e., stimulus condition) for each participant, for each time point at each electrode. These subject means are then averaged within each participant group to generate the grand mean waveforms for each stimulus condition at each electrode and are also plotted across time relative to the onset of the stimulus.

Statistical analyses of ERP data from sentence processing research seek to compare sets of individual mean waveforms from several different electrodes, in two or more different stimulus conditions, usually for two or more participant groups. This is most often accomplished via the mean amplitude method (although others are available), which involves the calculation of the mean EEG values across select time periods of interest, such as 300–500 ms poststimulus for an N400 and 500–800 ms poststimulus for a P600 (Tanner et al., Reference Tanner, Osterhout, Herschensohn, Chandlee, Franchini, Lord and Rheiner2009). After the mean amplitude is calculated for each participant across each time window, at each electrode, and in each stimulus condition, the means are compared statistically with a separate ANOVA for each time window. Thus, a typical ANOVA may include stimulus condition and electrode as within-subjects variables and group as a between-subjects variable.

Finally, as with self-paced reading and eye-tracking, ERP sentence processing experiments typically yield additional data from an offline distractor task. In ERP research, this is referred to as the behavioral data, which stands in contrast with the neural ERP data, and it most often takes the form of an acceptability judgment of the stimuli. The mean accuracy scores of participants are submitted to ANOVAs to determine whether there were any differences, especially between groups. Any effects or differences in the behavioral data can then be compared to those found in the primary ERP data, particularly when discussing and interpreting the overall outcome of a study (e.g., Tokowicz & MacWhinney, Reference Tokowicz and MacWhinney2005).