INTRODUCTION
Reading is not only an important comprehension skill that most second language (L2) learners strive to develop but also a major source of comprehensible input for L2 acquisition to occur. Hence, L2 reading activities are often designed with the dual aim of promoting development in comprehension ability and fostering the acquisition of L2 knowledge. To achieve these goals, it is necessary to explore and identify factors that affect the L2 reading process and the learning that accrues from engaging in reading activities. The characteristics of the reading activity is one variable that is likely to influence the nature of text processing as well as acquisition resulting from reading. While a few studies exist that investigate the influence of types of reading activity on text processing (e.g., Horiba, Reference Horiba2000, Reference Horiba2013; Kaakinen & Hyönä, Reference Kaakinen and Hyönä2005), little research has explored how the characteristics of reading activities may simultaneously affect the L2 reading process and noticing of L2 features. Given that reading activities are indispensable components of L2 instruction not only as means of developing reading skills but also as carriers of input, it is vital to begin filling this gap in the literature.
Against this background, the aim of this study was twofold. First, we intended to investigate how manipulating the characteristics of a reading activity, while keeping textual input constant, may influence reading processes. In particular, we aimed to explore whether activities designed to elicit more or less careful reading would indeed result in differential reading processes. Second, our goal was to examine how the extent of careful reading required by an activity may affect noticing and hence the possibility of learning L2 lexical and grammatical constructions from glosses. Glosses were operationalized as translations of linguistic items in the margin of the reading input. In broader terms, we hoped to explore the extent to which reading activity characteristics can facilitate opportunities for learning to read and reading to learn (Han & D’Angelo, Reference Han, D’Angelo, Han and J Anderson2009), similar to how manipulating the cognitive demands of productive communicative activities have been observed to enhance opportunities for developing new (e.g., Révész, Reference Révész2009) and practicing existing (e.g., Plonsky & Kim, Reference Plonsky and Kim2016) L2 knowledge and skills. The methodological novelty of our research lay in triangulating eye-tracking with stimulated recall data, a combination that is also just beginning to be utilized in the wider field of L2 research (Brunfaut & McCray, Reference Brunfaut and McCray2015; Smith, Reference Smith2012).
LITERATURE REVIEW
READING ACTIVITY CHARACTERISTICS AND L2 READING PROCESSES
Khalifa and Weir’s (Reference Khalifa and Weir2009) cognitive model for reading comprehension was considered as an ideal theoretical basis for the present study, given that this framework views reading as a cognitive process that constantly reacts to the reader’s goal, which, in turn, is expected to be influenced by the characteristics of the activity in which the reader is engaged. Khalifa and Weir (Reference Khalifa and Weir2009) presuppose three knowledge sources: the knowledge base, the central core, and metacognitive activity. The knowledge base subsumes the reader’s general world knowledge, topic knowledge, text-related knowledge, and linguistic knowledge. The central core entails a hierarchical system of lower-level and higher-level reading processes. The lower-level processes include word recognition, lexical access, syntactic parsing, and establishing propositional meaning, whereas higher-order processes comprise inferencing, building a mental model, and creating a text-level and intertextual representation. Metacognitive activity is concerned with setting goals, monitoring, and remediation.
Goal setting seems particularly relevant when considering the effects of reading activity characteristics on reading processes. As part of goal setting, the reader determines the type of reading required to perform an activity. This process could lead to careful or expeditious reading, taking place at either local or global level. Local reading entails extracting propositions at the clause or sentence level, whereas global comprehension involves understanding the text. The aim of careful reading is to process all the information in the text to achieve complete comprehension. By contrast, expeditious reading is quick and selective with the goal of identifying specific information, as in skimming or scanning. Thus, the goal setter enables readers to call upon different reading strategies and skills in response to differential activity demands.
Only a handful of studies exist that have investigated the relationship between reading activity characteristics and L2 reading processes, when textual difficulty was kept constant. In Horiba (Reference Horiba2000), the participants verbalized their thoughts while they read either freely as they normally would or for coherence by additionally paying attention to sentence relations. Activity type did not emerge as a predictor of comprehension, operationalized as content recall. However, more think-aloud comments referred to relational (i.e., relating textual information to background knowledge) and integrative processing (i.e., processing relations between sentences) when participants read for coherence. In another study by Horiba (Reference Horiba2013), participants were assigned to one of three conditions: reading for understanding new expressions, visualizing situations, and evaluating the author’s views. While reading comprehension was not found to vary across reading conditions, a follow-up think-aloud study revealed that lower-level processes were more prevalent when participants read for understanding new expressions, whereas higher-level operations were more frequent when they read for evaluation. These findings, overall, led Horiba to conclude that the effects of reading conditions may materialize more clearly in reading processes than outcomes. This conclusion, however, needs further verification.
The link between reading activity characteristics and L2 reading processes have received more attention in language testing. A study by Brunfaut and McCray (Reference Brunfaut and McCray2015) is of particular relevance here. This study, like the present research, examined the cognitive processes of test takers by triangulating stimulated recall data with eye-tracking. The participants were 25 test takers who performed four types of reading assessments (multiple-choice gap filling, sentence reordering, banked gap filling, and matching headings). Gap-fill items elicited more careful local reading and lower-level processing, while sentence ordering and matching headings involved proportionately more careful global reading, higher-level processing, and some expeditious reading. As evinced in previous studies investigating L1 (e.g., Kaakinen & Hyönä, Reference Kaakinen and Hyönä2005) and L2 (e.g., Bax, Reference Bax2013) reading processes, the authors found that the eye-movement metrics and stimulated recall protocols yielded converging but complimentary findings. The eye-movement analyses provided more insights into lower-level processes, whereas the stimulated recalls generated more information about higher-level processes.
To sum up, previous research indicates that L2 reading processes may vary according to reading activity characteristics. So far, however, researchers have primarily focused on the effects of reading activity types; little research has investigated how manipulating the characteristics of the same reading activity type might influence L2 reading. Triangulating eye-tracking with verbal protocol analyses appears a suitable methodological approach to address this research gap.
GLOSSING AND NOTICING OF L2 CONSTRUCTIONS
The impact of reading activity characteristics on glossing has received even less attention than the link between reading activity manipulations and L2 reading. This also constitutes an important gap in the literature because glossing (i.e., providing contextualized information about linguistic items in the form of definitions, synonyms, or translations) is a way to infuse focus on form into reading instruction. As a means of focus on form, glosses can arguably induce noticing (Leow, Reference Leow, Han and Anderson2009), defined by Schmidt (Reference Schmidt1990) as focal attention directed to linguistic elements accompanied with a low level of awareness. That is, glosses are assumed to have the capacity to trigger both attention and awareness, the two processes associated with noticing (Godfroid, Boers, & Housen, 2013). Input converted into noticed intake (Leow, Reference Leow2015), in turn, can potentially be rehearsed in working memory and made available for further processing that may eventually lead to L2 development (Leow, Reference Leow2015; Schmidt, Reference Schmidt1990). Thus, reading activities incorporating glosses can create what are considered favorable conditions for encoding new L2 representations by combining a primary focus on meaning with timely opportunities for noticing and processing of L2 form-meaning connections (Long & Robinson, Reference Long, Robinson, Doughty and Williams1998).
Given that noticing, by definition, entails focal attention allocated to linguistic information (Schmidt, Reference Schmidt1990), glossed linguistic information may be more prone to noticing when a reading activity, as in the present research, necessitates more careful reading. The more intensive and attentive processing of the texts will probably direct learners’ attention to the glosses and target linguistic items more frequently. Also, the more accurate understanding needed is expected to lead to more in-depth processing of the targeted constructions. In a similar vein, Robinson (Reference Robinson2001) suggested that more cognitively demanding tasks along certain dimensions would lead learners to seek more help from the input, resulting in deeper processing of input made salient through focus on form.
Although no research has examined the impact of manipulating the characteristics of reading activities on learner noticing of glosses, previous research has shown that, overall, glossed texts have positive, though small, effects on L2 development (e.g., Hulstijn, Hollander, & Greidanus, Reference Hulstijn, Hollander and Greidanus1996; Watanabe, Reference Watanabe1997). A small number of studies (Bowles, Reference Bowles2004; Guidi, Reference Guidi2009; Martinez-Fernández, Reference Martinez-Fernández2010) also investigated the impact of glossing on the noticing of L2 constructions using think-aloud protocols. Bowles (Reference Bowles2004) examined the extent to which computer-based versus paper-and-pen L1-glosses promoted awareness of glossed lexical items as compared to lack of exposure to glosses. The think-aloud comments revealed that the target vocabulary items were noticed more in the glossed conditions, but only negligible difference was observed between the computer and paper-and-pen groups. In Guidi (Reference Guidi2009), participants read either unglossed or L1-glossed texts, and glosses were provided to target lexical items and two grammatical constructions. No difference was found in the amount of reported awareness between the gloss and no gloss groups. In Martinez-Fernández (Reference Martinez-Fernández2010), participants were assigned to an L1 translation gloss, L1 translation fill-in-the blank gloss, or no gloss condition. The target features were lexical items and a grammatical feature. As in Bowles (Reference Bowles2004), participants reported awareness of more target lexical items in the gloss groups, but noticing was not affected by glossing type. The think-aloud protocols, however, revealed no effects of glossing for the noticing of the grammatical construction, similar to Guidi’s findings.
Given the small amount of research available and conflicting findings, more research is needed to explore the extent to which glosses can facilitate noticing during reading. As mentioned previously, further research is also warranted to examine how the characteristics of the reading activity may influence this link. Subsequent research would particularly benefit from utilizing verbal protocol data together with eye-tracking. While verbal protocols are suitable for tapping level of awareness (Gass & Mackey, Reference Gass and Mackey2017), eye-tracking is presumed to be a measure of attention (Reichle, Pollatsek, & Rayner, Reference Reichle, Pollatsek and Rayner2006). This combination, therefore, would allow for gauging both the quality (level of awareness) and quantity of attention (Godfroid et al., Reference Godfroid, Boers and Housen2013) triggered by glosses.
RESEARCH QUESTIONS
1. To what extent does manipulating the characteristic of an L2 reading activity, while keeping textual input constant, affect reading processes, as reflected in participants’ eye movements and stimulated recall comments?
2. To what extent does manipulating the characteristic of an L2 reading activity, while keeping textual input constant, affect the noticing of glossed linguistic constructions, as reflected in participants’ eye movements and stimulated recall comments?
The reading activity manipulation in the present study involved creating two versions of a text-ordering activity, which were designed to require more or less careful reading.
METHODOLOGY
DESIGN
Thirty-eight L2 users of English participated in the study. They completed two versions of a reading activity (henceforth, Text 1 and Text 2). Two types of target constructions, English unaccusative verbs and pseudolexical items, were glossed for both Text 1 and Text 2. Following a 2 × 2 repeated-measures design, participants were exposed to the two texts under a more careful or less careful reading condition (see Table 1).
Group assignment
Text order was counterbalanced across participants (see Figure 1). All participants took part in one session. They first completed a background questionnaire. Then, a pretest (a grammaticality judgment test, or GJT) was administered, followed by a proficiency test. While participants were carrying out the reading activities, their eye movements were recorded. Each reading was immediately followed by a short postreading questionnaire. Eleven participants were further invited to partake in a stimulated recall session right after completing the two reading activities and questionnaires. These students were randomly selected from among the participants who completed both activity versions. This allowed for comparing reading processes and noticing of glossed constructions under the two reading conditions. Finally, participants were administered an exit questionnaire.
Procedure of the study
PARTICIPANTS
The 38 participating students were native speakers of Korean, enrolled at a UK university. Thirty-two students were female, and the mean age was 27.84 (SD = 4.52). The average length of stay in an English-speaking country was 9.92 months (SD = 3.84). The participants had IELTS scores 6.5 (borderline CEFR B2/C1) or higher. To ensure homogeneity of proficiency among participants across the text and activity combinations, all students were administered an adapted version of the “Use of English” section of a practice Cambridge Proficiency English (CPE) test. Cronbach’s alpha for the CPE scores was .82.
TARGET CONSTRUCTIONS
We selected English unaccusativity as the target grammatical construction because this feature poses persistent difficulty for Korean learners (Chung, Reference Chung2014). Pseudowords were included to control for prior lexical knowledge.
English Unaccusative Verbs
Intransitive verbs can be classified into unergatives (e.g., Mary danced.) and unaccusatives (e.g., The snow melted.) (Perlmutter, Reference Perlmutter1978). An unergative verb assigns an agent role to its subject, where the agent/subject has a deliberate involvement in the event. The subject of an unaccusative verb lacks volitional control and performs a patient role. Researchers (e.g., Zobl, Reference Zobl, Gass and Schachter1989) have found that even high-proficiency learners tend to overpassivize unaccusatives (e.g., My mother was died when I was just a baby; ibid.). As Table 2 shows, 15 English unaccusative verbs were identified in the two treatment texts and selected as target constructions.
Target unaccusative verbs
Pseudowords
Both texts included five pseudowords (see Table 3). They were all nouns and appeared once in the texts. They substituted 10 original lexical items and followed English orthographic and morphological rules. When the original word was in the plural, the plural marker -s was retained. Each pseudoword consisted of seven letters, containing two syllables.
Target pseudowords
TEXTS
The two texts were expository passages selected from past TOEFL tests. The texts were chosen based on (a) whether they contained sufficient unaccusative verbs and (b) whether they covered topics likely to be unfamiliar to the participants. Text 1 was about petroleum resources, and the topic of Text 2 was the Cambrian period. The length of Text 1 and Text 2 were 682 and 699 words, respectively. Average readability, calculated from various indices (Flesch-Kincaid grade level, Gunning-Fog score, Coleman-Liau index, SMOG index, Automated Readability) was 11.6 for Text 1 and 13.4 for Text 2. These values indicated that the texts required at least upper-intermediate proficiency and thus were considered appropriate for the participants, who had at least low advanced proficiency.
READING ACTIVITY MANIPULATION
Both texts were divided into five segments. Each segment was presented on one page, following the original TOEFL format. The reading activities involved ordering parts of the segments (henceforth, text-ordering activity) and then answering multiple-choice comprehension questions (henceforth, reading comprehension test). The reading comprehension items were taken from the TOEFL tests, whereas the text-ordering component was added as part of the experimental manipulation.
Under the less careful reading condition, each text segment was split into two subparts, whereas, under the more careful reading condition, the segments were divided into three or four. The participants were asked to determine the correct order of the parts under both reading conditions. The participants were instructed to point and click the capital letter labeling each subpart in the order they considered correct (see Figure 2). We assumed that the version that required the reordering of more subparts would require more careful reading at both a local and global level (Khalifa & Weir, Reference Khalifa and Weir2009) due to the decreased clarity and coherence of text structure (Meyer & Ray, Reference Meyer and Ray2011).
An example page with AOIs (enclosed in rectangles)
The comprehension questions asked participants to identify information, make inferences, understand rhetorical purpose, recognize vocabulary meaning, simplify/paraphrase a sentence, or select main ideas of the text (Educational Testing Service, 2012). There were nine multiple-choice comprehension items for each text, with one or two questions following each segment. The maximum comprehension score was 10 points for each text. Each item was worth 1 point, except for the last item, for which the total score was 2 points. This item required completing a summary by selecting several responses from a group of multiple-choice options.
Double-spaced Courier font was used to present the texts. Each target item was underlined, and a corresponding Korean translation was provided in a marginal gloss. The participants were given 25 minutes for completing the activities. Piloting revealed that this time was sufficient to carry out the activities but put some pressure on participants under the more careful reading condition.
PRETEST
To measure the participants’ prior knowledge of unaccusativity, an untimed GJT was used as a pretest. The test included 15 grammatical and 15 ungrammatical sentences for the target unaccusative verbs (e.g., The sun was soon disappeared vs. The tension soon disappeared), and another 15 grammatical and 15 ungrammatical sentences served as distracters. The participants were asked to make binary choices (correct vs. incorrect). The maximum score was 30, and the test took approximately 7 minutes. The Cronbach’s alpha for the GJT was .62. The relatively low reliability might have been because participants showed little knowledge of unaccusativity, as reflected in their close to chance GJT performance on average (see Prior Knowledge of Unaccusatives section).
STIMULATED RECALL
After completing both reading activities, 11 students were asked to participate in a stimulated recall session prompted by their eye-gaze recordings. It was first explained to the participants in everyday language that the red circles and lines in the recordings indicated their eye fixations and saccades, respectively. They were also instructed to stop the recording at any time they wanted to verbalize what they were thinking while engaged in the original activity. The researchers also interrupted the recordings and prompted the participants to describe their thoughts during the performance of the reading activity on the few occasions when unusual or interesting eye movements were observed (longer fixations, regressive eye movements, or rereading behaviors), but these behaviors were not commented on by the participants. The stimulated recall sessions were video-recorded to capture participants’ spatial movements as well. Piloting revealed that participants often pointed at the computer monitor during the protocols (e.g., I started here, like this (pointing at screen), and it was very difficult.) The interviews were carried out in Korean.
QUESTIONNAIRES
The background questionnaire elicited information about the participants’ demographics and English language learning experience. The postreading questionnaire included two Likert-scale items gauging the participants’ familiarity with the reading topics. The exit questionnaire asked the participants to provide comments about their experiences during the reading activity. All questionnaires were administered in Korean.
PROCEDURE
Participants’ eye movements during the reading activities (i.e., text ordering and answering reading comprehension questions) were captured with a mobile Tobii X2-30 eye-tracker. Participants carried out the activities individually in a quiet room at the researchers’ institution while one of the researchers sat at a discrete distance to avoid any disruption caused by her presence. The sessions took approximately two hours for the nonstimulated recall and three hours for the stimulated recall students. To decrease participant fatigue, students were offered a break at several points in the experiment (after completing the pretest, proficiency test, and postreading questionnaires).
DATA ANALYSES
Eye-Movement Data
The eye-tracking data were analyzed with Tobii Studio 3.0.9 and the R statistical package. To assess the effects of reading condition on L2 reading processes (RQ1), two types of areas of interest (AOIs) were defined: (a) the text and (b) the text and response options combined (see Figure 2). AOIs for the texts were used in extracting indices associated with text-reading processes, reflecting participants’ comprehension processes. AOIs for the text and response options combined served as the basis for calculating measures of global reading processes, that is, they were presumed to shed light on how participants coped with the activity. Then, inspired by Brunfaut and McCray (Reference Brunfaut and McCray2015), 10 indices of text and global processing were calculated based on the eye-fixation and saccade data with a series of R-scripts (McCray, Reference McCray2016). The measures are summarized in Table 4. For each index, we expected greater values under the more careful reading condition, as ordering of the text subparts would likely require more intensive and recursive text processing. The only exception was median forward saccade length. For this measure, we predicted smaller values in the activity requiring more careful reading, as textual processing would probably be more frequently interrupted.
Eye-movement measures
Next, to examine if the reading activity manipulation affected participants’ processing of glossed constructions (RQ2), AOIs were defined for each target feature and gloss. While the target areas were identical in pixel size for pseudowords and glosses, those for unaccusative verbs were inevitably dissimilar due to the different verb lengths. This did not confound the results, as both versions of the activity included the same AOIs. Eight eye-tracking measures were extracted for the target constructions and glosses (see Table 4). We hypothesized that, under the more careful reading condition, the values for each measure would be higher, as more thorough text processing would result in more attention to the target words and their glosses.
Statistical Analyses
Prior to any statistical procedures, we performed a power analysis for all tests using GPower 3.1 (Faul, Erdfelder, Lang, & Buchner, Reference Faul, Erdfelder, Lang and Buchner2007). The sample size was found to be adequate to detect medium effect sizes for all factors of interest with an α = .05 and power = .90. The Statistical Package for the Social Sciences (SPSS) 22 was used to compute reliability and descriptive statistics. The rest of the analyses were conducted with version 3.3.0 of the R statistical package, by constructing mixed-effects models using the lmer function of the lme4 package. We first established that, across the two versions of the activity, participants were equivalent in English proficiency and pretest scores, and that topic familiarity had no confounding influence on reading comprehension. It was also confirmed that there were significant differences in perceived mental effort between the two activity versions.
Then, the research questions were addressed, with each mixed-effects model including an eye-tracking measure as the dependent variable. In all models, Condition (i.e., reading activity version), Text, and their Interaction were the fixed effects (i.e., independent variables), and Subject was the source of random effects. The modeling started by constructing null models that contained only a random intercept for Subject. Next, Condition, Text, and the Interaction were added, and it was tested whether their addition improved the fit of the null models. In this step, likelihood ratio tests were conducted using χ 2 statistics. If a significant fixed effect was identified, a maximal random effects structure was produced to examine the magnitude and direction of the fixed effect(s) on the eye-movement measure. Given that maximal random structures can be overly complex with multiple random slope parameters, models often fail to converge. If this was the case, the random effect parameters accounting for the least variance were removed one by one until convergence was achieved. As participants were assigned to the reading conditions following a 2 × 2 repeated measures design, when Condition emerged as a significant factor in the likelihood ratio tests, multilevel mixed-effects models were developed including a within-subject random slope for Condition. A within-subject random slope for Text was not included in the models, as participants produced only one value, that is, either an eye-movement index or time taken to complete the activity, for each text.
An absolute t-value above 2.0 was the criterion for significance. Effect sizes were computed with the r.squared GLMM function from the MuMln package. R 2 values above .06, .16, and .36 were considered as small, medium, and large, respectively (Plonsky & Oswald, Reference Plonsky and Oswald2014). Collinearity statistics for the fixed effects (Condition and Text) were calculated using the collin.fnc function in the languageR package. Following Baayen (Reference Baayen2008), condition numbers between 0 and 6 were regarded as evidence for no collinearity, around 15 as medium collinearity, and 30 or above as potentially harmful collinearity.
Stimulated Recalls
The stimulated recalls were transcribed using the video-transcription software F5, version 2.2. The transcripts were uploaded to NVivo 10.0.3 for qualitative analysis. The first author reviewed the transcripts and identified emergent categories by annotating the data (for the resulting coding scheme, see Table 9). Then, a randomly selected subset of the video-recordings (13.6%) was watched and coded by a second coder to verify coding reliability. Inter-coder agreement was 90 per cent with a kappa of .71, which was acceptable, SE = 1.02, 95% CI [- .98, 3.06]. Next, the comments were further categorized depending on whether they concerned the reading conditions, and frequency counts were calculated for each code under each condition.
RESULTS
PRELIMINARY ANALYSES
English Proficiency
The CPE scores were analyzed to check if the participants, across reading conditions and texts, were homogeneous in their proficiency (see Table 5 for descriptive statistics). The likelihood ratio tests revealed that the null model was not improved by adding Condition (χ 2(1) = .01, p = .93, R 2 < .01), Text (χ 2(1) = .01, p = .99, R 2 < .01), or the Interaction (χ 2(1) < .01, p = .99, R 2 < .01). In other words, there was no significant difference in proficiency among the participants depending on text or reading condition.
Descriptive statistics for proficiency test
Maximum score = 30. Each student’s score is included twice due to the design.
Prior Knowledge of Unaccusatives
The GJT scores were analyzed to test whether the participants had parallel prior knowledge of unaccusatives across the experimental conditions. The maximum score was, again, 30. Under the less careful condition, the mean score of the participants who read Text 1 and Text 2 were 15.74 (n = 19, 95% CI [14.84, 16.64], SD = 4.57) and 15.39 (n = 19, 95% CI [14.49, 16.29], SD = 4.36), respectively. For the version requiring less careful reading, the mean was 14.95 (n = 19, 95% CI [14.05, 15.85], SD = 3.52) for Text 1 and 15.53 (n = 19, 95% CI [14.63, 16.43], SD = 4.02) for Text 2. The likelihood ratio tests constructed with the pretest GJT scores indicated that adding Condition (χ 2(1) = .01, p = .93, R 2 < .01), Text (χ 2(1) = .01, p = .99, R 2 < .01), and Interaction (χ 2(1) < .01, p = .99, R 2 < .01) did not improve the null models. That is, at the pretest, the participants did not differ in their ability to judge the grammaticality of unaccusative sentences across text and reading activity allocation.
Topic Familiarity
To assess whether topic knowledge affected participants’ comprehension, topic familiarity was measured using post-reading Likert scale items (Item 1: I thought this topic of the reading was familiar; Item 2: I had some background knowledge about the reading topic.). The maximum value for each item was 7.00. The responses to the items correlated significantly, Text 1: r(38) = .80, p < .01, Text 2: r(38) = .87, p < .01, suggesting that the items assessed overlapping constructs. For participants who performed the version designed to elicit less careful reading, the mean value was 2.62 for Text 1 (n = 19, 95% CI [1.93, 3.31], SD = 1.54) and 1.87 for Text 2 (n = 19, 95% CI [1.50, 2.24], SD = .83), whereas the mean under the more careful reading condition was 2.05 for Text 1 (n = 19, 95% CI [1.60, 2.50], SD = 1.01) and 2.23 for Text 2 (n = 19, 95% CI [1.60, 2.86], SD = 1.39). Likelihood ratio tests found that adding Familiarity and its interactions with the fixed effects did not improve the null model, Familiarity (χ 2(1) = 1.56, p = .21, R 2 < .01), Condition*Familiarity (χ 2(1) = .89, p = .34, R 2 < .01), Text*Familiarity (χ 2(1) = 1.47, p = .22, R 2 < .01), and Familiarity*Condition *Text (χ 2(4) = 5.37, p = .25, R 2 = .01). That is, topic familiarity did not affect reading comprehension across the two activity versions or texts.
Text-Ordering Performance and Reading Comprehension Scores
Table 6 presents the descriptive statistics for participants’ text-ordering activity performance and comprehension test scores. Likelihood ratio tests revealed that both Condition (χ 2(1) = 74.60, p < .01, R 2 = .19) and Text (χ 2(1) = 12.56, p < .01, R 2 = .03) had a significant influence on text-ordering performance, but not their Interaction (χ 2(1) = .23, p = .63, R 2 = .02). The effect size was medium for Condition and small for Text. According to the maximal-structure models, text-ordering performance was significantly better for the less careful reading conditions (Estimate = .44, t = 8.85) and for Text 2 (Estimate = .20, t = 3.79). Another set of likelihood ratio tests indicated that Condition (χ 2(1) = 1.47, p = .22, R 2 = .02), Text (χ 2(1) < .01, p = .94, R 2 = .02), or the Interaction (χ 2(1) = .65, p = .42, R 2 < .01) did not have an influence on the comprehension scores.
Descriptive statistics for text-ordering and reading comprehension scores
The maximum score: text ordering = 5.00; reading comprehension test = 10.00.
Time Taken for Activity Completion
The mean time under the more careful reading condition was 835.45 (n = 38, 95% CI [818.19, 819.47], SD = 203.93) and 757.43 (n = 38, 95% CI [818.19, 819.47], SD = 203.93) under the less careful reading condition. A likelihood ratio test yielded a significant difference between the time taken to complete the two versions of the activity (χ 2(1) = 9.67, p < .01, R 2 = 06). The summary of a maximal-structure model demonstrated that it took significantly longer to complete the versions intended to elicit more careful reading (Estimate = 102.52, t = 3.30).
Next, we analyzed whether time on activity differed by text version. Participants on average took 774.04 seconds to complete the activities based on Text 1 (n = 38, 95% CI [773.40, 774.68], SD = 192.44) and 818.83 seconds based on Text 2 (n = 38, 95% CI [818.19, 819.47], SD = 203.93). A likelihood ratio test revealed that there was no significant difference between these times (χ 2(1) = 2.79, p = .09, R 2 = .01).
EYE-MOVEMENT DATA
Reading Activity Characteristics, Eye-Gaze Behaviors, and Reading Processes
Table 7 presents the descriptive statistics for the eye-movement measures computed to investigate reading processes (RQ1). From a series of likelihood ratio tests, Condition emerged as a significant predictor for the following measures: number of fixations for texts and responses combined (χ 2(1) = 13.39, p < .01, R 2 = .12), number of fixations for texts only (χ 2(1) = 29.46, p < .01, R 2 = .27), sum of fixation durations for texts only (χ 2(1) = 16.91, p < .01, R 2 = .17), number of forward saccades (χ 2(1) = 23.25, p < .01, R 2 = .21), and number of regressions (χ 2(1) = 18.26, p < .01, R 2 = .16). The interaction of Condition and Text also improved the null models for the sum of fixation durations for texts and responses combined (χ 2(1) = 13.24, p < .01, R 2 = .22) and proportion of regressions (χ 2(1) = 4.17, p = .04, R 2 = .03). No significant effects were observed for the rest of the measures.
Descriptive statistics for eye-movement measures of reading processes
As shown in Table 8, post hoc multilevel mixed-effects models confirmed that the versions designed to promote more careful reading generated more eye fixations for the activity (text and responses combined) and the text segments only. Participants also fixated longer on the texts and made significantly more forward saccades and regressive eye movements under the more careful reading condition. That is, as intended, participants appeared to engage in more careful and recursive reading, as manifested in the longer overall eye-gaze duration and increased number of fixations on the texts, and more forward saccades and regressions. Turning to the interactions, the more careful reading condition led to an increase in the sum of fixation durations on both text and response sections in Text 2 but decreased the index in Text 1. Also, the condition created to elicit more careful reading resulted in greater proportion of regressions in Text 1, but lower in Text 2. The R 2 values for these relationships ranged from .12 to .22, indicating small-to-medium effect sizes. The only exception was a very small effect size (R 2 = .03) for the Interaction on the proportion of regressions.
Summary of mixed-effects models for eye-movement measures of reading processes
Significance: | t | > 2.0.
Reading Activity Characteristics, Eye-Gaze Behaviors, and Noticing
Table 9 presents the descriptive statistics for the eye-movement measures calculated to examine the effects of reading activity manipulation and text version on the noticing of glosses and glossed constructions. The likelihood ratio tests indicated that both Condition and Text significantly improved the model fit for the number of fixations (Condition: χ 2(1) = 20.44, p < .01, R 2 = .23, Text: χ 2(1) = 7.88, p < .01, R 2 = .09) and sum of fixation durations (Condition: χ 2(1) = 12.50, p < .01, R 2 = .15, Text: χ 2(1) = 10.89, p < .01, R 2 = .12) on the target unaccusative verbs. Interaction between Condition and Text, however, did not improve model fit (number of fixations: χ 2(1) = .97, p = .32, R 2 = .34, sum fixation durations: χ 2(1) < .01, p = .93, R 2 = .28). No significant effects were found for the rest of the indices.
Descriptive statistics for eye-movement measures of noticing
As Table 10 illustrates, post hoc multilevel mixed-effects models confirmed Condition and Text as significant predictors of the number and sum of fixations on the target unaccusative verbs. The participants fixated significantly more often and longer on the target verbs when performing the versions constructed to promote more careful reading and when working on Text 2. The effect sizes were medium for both number of fixations (R 2 = .34) and sum of fixations (R 2 =.28).
Summary of mixed-effects models for eye-movement measures of noticing
Significance: | t | > 2.0.
STIMULATED RECALL PROTOCOLS
Eight meta-codes emerged from the stimulated recall comments: high difficulty, low difficulty, ability to concentrate on activity, comprehension, word-level cue, discourse-level cue, noticing target unaccusative verbs, and noticing target pseudowords. Each meta-code was broken down into subcodes (see Table 11 for examples). More annotations were counted for the more careful (n = 374) than the less careful reading conditions (n = 230) overall and for most individual codes. When participants described their performance under the more careful reading condition, they more frequently reported experiencing difficulty and feeling unconfident. References to certain reading strategies, such as careful reading, skimming, and searching for hints, were also more frequent among comments on the version of the activity, which was designed to elicit more careful reading. Likewise, participants reported relying on linguistic cues more often when describing their thoughts under the more careful reading condition: At the word level, the participants remembered utilizing keywords, pronouns, and transitional words more frequently; at the discourse level, there were more comments indicating that the participants struggled to order the text segments. They also reported with greater frequency that they focused on the first and the final sentence of each text segment and coherence between sentences. Lastly, although few comments concerned the target items and the glosses, there were more comments referring to the unaccusative verbs and their glosses when participants recalled their thoughts during the activities created to facilitate more careful reading.
Code frequency for stimulated recall comments (n = 604)
For a few codes, however, recalls for the less careful reading condition yielded more comments. For example, the participants remembered rereading the texts more frequently during this activity version. In addition, they reported focusing more on articles, first mention of words, and sentence connectives. Finally, among the few comments generated about the glosses, participants referred to noticing glosses for pseudowords more often when recalling their performance on the activity constructed to promote less careful reading.
DISCUSSION
The present study investigated the effects of manipulating the characteristics of reading activities on L2 reading processes and noticing of glossed linguistic constructions. The reading activity involved ordering jumbled texts, in which paragraphs were split into two (less careful reading condition) versus three to four (more careful reading condition) segments.
READING ACTIVITY CHARACTERISTICS AND L2 READING PROCESSES
Our first research question asked the extent to which manipulating the characteristics of L2 reading activities affected reading processes, as reflected in participants’ eye movements and stimulated recall comments. We hypothesized that the need to reorder more subparts would prompt more careful reading at both the local and global level (Khalifa & Weir, Reference Khalifa and Weir2009), given the more intensive and attentive reading required to identify intersentential relations (Meyer & Ray, Reference Meyer and Ray2011). As predicted, both the eye movement and stimulated recall data confirmed that the version designed to elicit more careful reading, as intended, generated more thorough and intensive text processing. When performing the versions constructed to elicit less careful reading, the participants fixated more frequently on the activity (text and response options combined) and more often and longer on the texts. Participants also produced a larger number of forward saccades and regressive eye movements, indicating that they engaged in more attentive and recursive text processing. The effect sizes ranged from small to medium. The stimulated recall comments revealed that, under the more careful reading condition, participants more frequently employed certain reading strategies, such as skimming, careful reading, and searching for hints. They also recalled more extensive use of lexical and discourse cues. That is, the eye-tracking and stimulated recall data seem to converge and confirm that, when carrying out the activity versions created to promote more careful reading, participants indeed processed the texts more carefully and intensively.
It is also important to point out, however, that for some of the eye-movement measures, no significant difference (median fixation duration, median forward saccade length, median regression length, and proportion of regressions) or an interaction effect (sum of fixation durations for text and response options combined and proportion of regressive movements) was found between the two reading conditions, contrary to our expectations. A possible explanation may be that, although the two versions of the activity led to a differential amount or quantity of processing, they did not prompt reading processes that were qualitatively different in nature. Unlike frequency and sum of duration measures, medians of fixation, saccade, and regression lengths are likely to capture qualitative differences in reading. For example, longer saccade lengths are probably more associated with global reading because global reading (e.g., reading for gist) necessitates less detailed comprehension (Brunfaut & McCray, Reference Brunfaut and McCray2015). On the other hand, shorter saccades are more likely to reflect engagement in local reading (e.g., reading for detail), requiring more thorough text processing. The same reason might explain that no considerable activity effects emerged for regressive movements (although a significant interaction was identified for this index, the effect size was small). Proportion of regressive movements is also likely to vary when readers engage in qualitatively different processes. A gap-fill activity with a given set of words, for instance, would likely involve more regressive movements than the text-ordering activity here, as readers would probably revisit the list of words on several occasions while working on the activity. The stimulated recall data also suggest that our reading activity manipulation had primarily quantitative effects on reading processes: participants recalled using certain strategies with greater frequency under the more careful reading condition, but rarely mentioned qualitatively different strategies.
Let us now attempt to explain the interaction identified for sum of fixation durations for text and responses combined, that is, why participants would fixate shorter overall under the more careful reading condition for Text 1, while the pattern was in the expected direction for Text 2 with longer overall fixation duration in the more careful reading condition. A possible clue is that participants achieved considerably lower mean scores on the text-ordering activity under the more careful reading condition for Text 1 (Mean = 1.00) than Text 2 (Mean = 1.95), suggesting that increasing the need to engage in careful reading resulted in proportionately greater demands for Text 1. This might have left less attention for answering the comprehension questions, which, in turn, might have led to shorter fixations on the Text 1 comprehension questions (but not the text). This account is consistent with the fact that, as discussed previously, sum of fixation durations for text only were, just as for Text 2, higher for the more careful reading condition for Text 1. Another possible explanation is that the more careful reading condition for Text 1, which appeared to be even more demanding than that for Text 2, encouraged participants to engage in the text more thoroughly and repeatedly, resulting in quicker completion of the comprehension items.
Finally, it is worth noting that the eye-gaze data yielded largely overlapping patterns for Text 1 and Text 2, only one measure yielded an interaction between text version and activity version. This suggests that the effects of the reading activity manipulation observed here might generalize to other academic expository texts. Clearly, future research is needed to confirm this and to test whether our results would transfer to other genres.
READING ACTIVITY MANIPULATION AND NOTICING GLOSSED LINGUISTIC CONSTRUCTIONS
Our second research question investigated the extent to which manipulating the characteristics of reading activities may influence the noticing of glossed linguistic constructions, as reflected in participants’ eye movements and stimulated recall comments. We expected that participants would notice glossed information to a greater degree under the more careful reading conditions, as the more intensive and attentive text processing required would direct attention to the glosses and target linguistic items more frequently. Also, the more precise understanding needed would promote more in-depth processing of the target form-meaning relationships. As hypothesized, the eye-movement indices revealed that the target unaccusative verbs received considerably more attention in the more careful reading condition, evidenced in the significantly greater number of fixations and longer fixation durations on the target verbs. The effect size for these relationships was large. Similarly, the stimulated recall data demonstrated that all the comments related to the target verbs (n = 3) concerned the more careful reading condition. It should be pointed out, however, that the number of comments on the target verbs was relatively small. Considering that the stimulated recall comments indicate a low level of awareness (Schmidt, Reference Schmidt1990) or depth of processing (Leow, Reference Leow2015), the findings appear to imply that the impact of the reading activity manipulation on learners’ awareness appeared to be only marginal. Taken together, the more careful reading condition was more likely to trigger attention to and awareness of the unaccusative verbs, but its impact seemed more pronounced on attention than awareness.
Interestingly, however, the reading activity manipulation did not affect the overall amount of attention paid to the glosses associated with the target verbs. That is, the increased need to engage in careful reading, according to the eye-movement data, did not encourage learners to check the glosses with greater frequency or process them longer. In fact, verb glosses were often ignored; the average number of fixations to all verb glosses was below 4 for both texts regardless of activity version although Text 1 and Text 2 included 8 and 7 target verbs, respectively. The participants also made few stimulated recall comments about the verb glosses, suggesting that they might have rarely been the focus of attention. This was probably because the unaccusative construction is of low communicative value. Therefore, if participants were familiar with the meaning of the root verb, they might have disregarded the glosses as the grammatical information in them was not essential to comprehension.
It was also contrary to our expectations that activity version had no impact on the noticing of pseudowords and their glosses, as indicated by a lack of a significant difference in the number and sum of eye fixations at pseudowords and their glosses across the reading conditions. One reason for this may be that the processing of pseudowords was less essential to the completion of the activity than that of the unaccusatives. If the pseudowords had been selected based on task essentialness, the difference in reading conditions might have affected the extent to which they were attended to and processed.
Finally, like for RQ1, the eye-movement data generated similar patterns for Text 1 and Text 2, implying that the results might generalize to other academic expository texts.
IMPLICATIONS
On the theoretical front, the stimulated recall results corroborated hypotheses deduced from Khalifa and Weir’s (Reference Khalifa and Weir2009) model of L2 reading, suggesting that, depending on reading activity characteristics, readers can call upon differential strategies and skills to achieve their goals. By theorizing the relationships between reader goals, metacognitive activities, and reading processes, Khalifa and Weir’s model would appear as a useful theoretical starting point for future work exploring links between reading activity characteristics and L2 reading. We also found some evidence for our hypothesis that increasing the need to engage in careful reading would generate greater attention and awareness of glossed linguistic constructions.
At the methodological level, we confirmed that combining eye-tracking with stimulated recall is a useful way to tap reading processes, enabling the investigation of both lower (e.g., saccades) and higher-order (e.g., strategies) reading operations. Triangulating eye-tracking and stimulated recall data is new to glossing research, and we found it helpful to assess both qualitative and quantitative aspects of attentional allocation (Godfroid et al., Reference Godfroid, Boers and Housen2013), and thereby gain information about the effects of reading activity characteristics not only on attention or awareness but also on both processes.
Finally, the findings of this study suggest that increasing the need to carry out careful reading, although having limited impact on awareness, may result in more attention to target L2 features, as shown in the case of the target unaccusative verbs. That said, a potential pedagogical implication is that manipulating reading activities in such a way that they elicit more careful reading can promote more attention to textually enhanced grammatical constructions, which may otherwise remain unattended. For example, a manipulation that induces learners to evaluate text coherence or structural organization would probably encourage more attentive text processing.
LIMITATIONS AND FUTURE RESEARCH
One limitation of this research lies in the use of the stimulated recall methodology. As stimulated recall involves a posteriori recollection of cognitive processes, it is possible that only a subset of the conscious processes during performance was reported. Another issue concerns the selection of target lexical items. We used two criteria: single occurrence in the text and being a noun. Additional factors that would ideally be considered in future research include concreteness, inferrability, and position in sentence. It would also be worthwhile to explore whether the activity manipulations would have a stronger impact on noticing if the processing of lexical items were made essential to the successful completion of the activity. In this study, the target lexical items appeared to have low functional load. A further weakness originates from the within-subjects design. Although this allowed controlling for individual differences, it made it impossible to investigate the combined effects of reading activity characteristics and glossing on L2 development. Adopting a between-subject design could address this limitation. The study would also have benefited from using a more high-precision eye-tracker, especially as regards the noticing measures. Compared to the reading indices, the AOIs for noticing were relatively small, thus more prone to error. Finally, this study utilized a single activity type, one type of reading activity manipulation, an academic expository text, and low-advanced/advanced Korean L2 readers. Future research should examine whether our results would extend to different activity types, activity manipulations, genres, proficiency levels, and L1 speakers.
CONCLUSION
This study aimed to launch a new line of research into how manipulating the characteristics of L2 reading activities may affect L2 reading processes and the noticing of glossed constructions while reading. In general, we hoped to help fill a gap in instructed SLA research on how the characteristics of reading activities may influence opportunities to learn to read and read to learn (Han & D’Angelo, Reference Han, D’Angelo, Han and J Anderson2009). A methodological innovation of our research was the triangulation of eye gaze with stimulated recall data. Reflecting a prediction that we derived from Khalifa and Weir’s (Reference Khalifa and Weir2009) reading model, we observed that manipulating a text-ordering activity resulted in more careful reading. As hypothesized, we also found that an increased need to engage in careful reading led to greater attention to and awareness of a glossed grammatical construction, with attention appearing to be more affected by the manipulation of reading activity characteristics than awareness. In contrast to what we expected, however, no relationship emerged between our manipulation and noticing of glossed pseudowords.
