BACKGROUND
A common way of facilitating learners’ text comprehension and supporting the intake of new vocabulary is to provide glosses that clarify the meaning of unfamiliar words. Glosses with multimodal content (i.e., both pictorial and textual clarifications) benefit intake of word meaning particularly well, according to postreading tests in some studies (e.g., Kost, Foss, & Lenzini, Reference Kost, Foss and Lenzini1999; Yoshii & Flaitz, Reference Yoshii and Flaitz2002). The reported benefits are often attributed to the advantages of coding information both verbally and nonverbally, in keeping with, for example, Paivio’s (Reference Paivio1986) dual coding theory and Mayer’s (Reference Mayer2009) framework of multimedia learning. These accounts imply that adding a pictorial elucidation of word meaning triggers processing of the word that is qualitatively different from processing engendered by a gloss with only a textual clarification.
However, Boers and colleagues (2017a) argue for an alternative, or at least complementary, account for the reported benefits of multimodal glosses, namely that they are less likely to be ignored than text-only glosses and also invite longer processing. However, the extent to which different glosses influence learners’ engagement with glossed words has not yet been properly investigated because previous studies only used offline measures of learning, not online measures of reading behavior. This is an unfortunate gap in the research, not only because of the theoretical debate, but also because better insights into how gloss types influence reading behavior may inform the design of pedagogic materials.
The present study is a step toward filling that gap. We recorded eye movements of adult learners of English as a Second Language (ESL) as they read a story containing six pseudowords each accompanied by a textual gloss, a pictorial gloss, or a multimodal gloss (i.e., comprising both the picture and the textual gloss). Unannounced tests gauged learners’ recall of the form of these target pseudowords and recognition of their meaning. So far, most studies of multimodal glossing have focused on learners’ retention of word meaning, but it is worth examining learners’ recall of orthographic form as well because gloss types may differently influence the attention given to form.
GLOSS TYPES AND VOCABULARY ACQUISITION FROM READING
Research on the effects of gloss types is wide-ranging and includes, for example, comparisons of first (L1) and second language (L2) glosses (Ko, Reference Ko2012) and ways of promoting cognitive engagement through multiple-choice formats (Watanabe, Reference Watanabe1997). Here, however, we focus on studies comparing the effect on L2 vocabulary uptake of glosses with and without pictorial components, mostly in incidental learning conditions.
As an example of research focusing primarily on the uptake of word meaning, Kost et al. (Reference Kost, Foss and Lenzini1999) compared three gloss conditions: text only (L1 translations of test words), picture only, and multimodal with both text and picture. Learners seeing multimodal glosses outperformed those seeing text-only glosses in a posttest requiring selection of the appropriate picture for each test word. This is unsurprising because participants in the multimodal condition had seen the pictures used in the test during their reading of the text, while those in the text-only condition had not. This test showed no significant difference between multimodal and picture-only conditions. A second posttest involved matching test words with L1 translations. Unsurprisingly again, performance was better if participants had seen the L1 text glosses than if they had only seen pictures. However, the multimodal condition produced significantly higher scores than either of the other conditions. So, seeing both the illustration and the L1 translation in the gloss appeared to produce better retention of word meaning than seeing just the translation.
Yoshii and Flaitz (Reference Yoshii and Flaitz2002) conducted a replication of Kost et al. (Reference Kost, Foss and Lenzini1999) in a multimedia context. In a picture-recognition posttest, participants in the multimodal condition again outperformed the text-only group, with no significant difference from the picture-only condition. As in Kost et al. (Reference Kost, Foss and Lenzini1999), picture-only glosses were relatively unhelpful when participants were asked to either supply or recognize definitions of the words. Overall, the trend was again for multimodal glosses to result in greater learning than either single gloss condition.
In another computer-based multimedia presentation of reading materials, Chun and Plass (Reference Chun and Plass1996), Plass et al. (Reference Plass, Chun, Mayer and Leutner1998), and Akbulut (Reference Akbulut2007) accompanied target words with textual and visual annotations. Participants indicated which annotation(s) to consult by clicking on hyperlinks. More correct posttest responses were given if both text and picture annotations had been consulted than just text annotations. Similar results were found by Jones and Plass (Reference Jones and Plass2002), using a reading-while-listening task where the participants could pause the recording and choose to click links to textual or pictorial word annotations. However, the results of these experiments may in part be due to the number rather than type of annotations consulted for each target word (Boers et al., Reference Boers, Warren, Grimshaw and Siyanova-Chanturia2017a).
While these studies show benefits of multimodal glosses for the retention of word meaning, there are some exceptions: Neither Acha (Reference Acha2009) nor Boers et al. (Reference Boers, Warren, He and Deconinck2017b) found evidence supporting multimodal over text-only glosses.
EYE-TRACKING STUDIES WITH A FOCUS ON VOCABULARY
The studies summarized in the preceding text, like most studies of vocabulary uptake from reading, employed postreading tests, which gauge the outcome of the reading process but have little to say about that process. To better understand the latter, in both L1 and L2, researchers have studied eye movements during reading. Eye-movement research (e.g., Rayner, Reference Rayner1998, Reference Rayner2009) uses measures such as visual fixation on a word as proxies for attention (see Godfroid and Schmidtke [Reference Godfroid, Schmidtke, Bergsleithner, Frota and Yoshioka2013] for a discussion of the relationship between visual fixation and the concepts of attention and awareness).
To date, few studies have researched eye-movement patterns in the context of incidental vocabulary acquisition during reading. These have shown that unfamiliar words attract more attention than familiar ones, that attention paid to novel words predicts their retention in memory, and that the increased attention declines over multiple encounters. We review here some of that evidence.
In an early study, Chaffin et al. (Reference Chaffin, Morris and Seely2001) investigated how readers establish the meanings of new words during silent L1 reading. Sentence pairs comprised a sentence containing the target word and a second sentence containing a related word. The first sentence was either neutral or highly informative concerning the target. Reading times for this first sentence depended on target familiarity (e.g., guitar vs. zither vs. the pseudoword asdor) and on the informativeness of the context. Crucially, readers spent more time on the related word in the second sentence (e.g., in this case instrument) when the target word was unknown and the context was uninformative (neutral). This suggests that readers successfully identify and pay attention to the portion of the text that is relevant for inferring the meaning of an unknown word, which, in the case of an otherwise uninformative context, was the related word. As the authors acknowledge, however, they collected no direct evidence that learning of the novel word’s meaning had taken place (though they cite supporting evidence from a previous study using the same materials and a direct assessment of understanding).
In a study of the effects of word familiarity in silent L1 reading, Williams and Morris (Reference Williams and Morris2004) measured both visual fixation patterns and learning outcomes. Both initial and second pass reading time (i.e., rereading) indicated that unfamiliar words received more attention. Postreading tests showed better meaning retention for many of these unfamiliar words. Interestingly, initial reading time was shorter and second pass reading time longer on unfamiliar words that received correct posttest responses than on those that received incorrect responses. Because the preceding sentence context was not highly informative for interpreting the unknown word, the authors interpret this behavioral pattern to show that readers do not dwell on novel words if they have little information to aid their interpretation but will return to them (as reflected in second pass reading times) if the following context gives them something to work with.
More recently, Brusnighan and Folk (Reference Brusnighan and Folk2012) investigated the role of morphological and contextual information in incidental vocabulary acquisition during L1 reading. Eye movements were measured as participants read English compounds that were either familiar and transparent (milkshake), novel and transparent (drinkblend), familiar and opaque (cocktail), or novel and opaque (deskdoor), each occurring in either neutral or informative sentence contexts. Posttests showed that their meanings could be retained after a single encounter, even for novel forms. As expected, novel items received longer reading times than familiar items. In addition, morphologically transparent novel items embedded in informative contexts (i.e., when both morphology and context cued the meaning of the item) had shorter rereading times, indicating a processing advantage.
The relationship between eye movements (as a proxy for attention) and pseudoword learning in short L2 passages was explored by Godfroid et al. (Reference Godfroid, Boers and Housen2013). Postreading recognition was measured by presenting learners with the sentence in which a pseudoword had been originally encountered, but with the pseudoword replaced by a dotted line, and requiring them to select the appropriate item from 18 candidate items, including the pseudoword and 11 other pseudowords used in the experiment. There was a positive correlation between fixation times on pseudowords and their recognition accuracy.
A range of posttests was employed by Pellicer-Sánchez (Reference Pellicer-Sánchez2016), including immediate and delayed tests of the acquisition of both word meaning and word form. Both L1 and proficient L2 speakers read stories for comprehension. The study also considered the effect of repetition, with novel words appearing multiple times in contexts deemed helpful for inferring meaning. The two reader groups differed in acquisition rate, but not in terms of outcome. After eight exposures to the novel words, L2 readers recognized their form and meaning in multiple-choice tests with 86 percent and 75 percent accuracy, respectively, and their success in supplying word meaning was 61 percent. The eye-movement data showed decreases in fixation counts and durations after three or four encounters. By the eighth encounter they were read similarly to known words. Again, greater total reading times (TRTs) on the novel words predicted better learning.
In a similar study investigating multiple encounters with novel words during story reading, Mohamed (Reference Mohamed2018) also found a gradual decrease in fixation durations as advanced L2 readers became more familiar with the words. The novel words varied in the number of times (1–30) they occurred in the story. Once again, total reading duration predicted learners’ performance on posttests of word meaning and form and was a stronger predictor than the number of instances of the novel words.
Finally, Elgort et al. (Reference Elgort, Brysbaert, Stevens and Van Assche2017) studied L2 readers’ eye movements as they read an expository text (chapters from a general academic book). Again, unfamiliar words occurred with varying frequencies (8–64 occurrences). Time spent on the words decreased, most markedly over the first 10 encounters. Nevertheless, even after as many as 40 encounters noticeable differences in fixations and reading times remained between target words and familiar control items. Participants’ mean success in supplying target word meanings in a posttest was 34 percent. Both findings contrast with those of Pellicer-Sánchez (Reference Pellicer-Sánchez2016), perhaps because of the helpful contextual cues that accompanied the target words in the latter.
Likely owing to the complex study design (e.g., the reading was spread over two days, introducing a longer time interval between word encounters and the posttest), Elgort et al. (Reference Elgort, Brysbaert, Stevens and Van Assche2017) did not report an association between TRTs and posttest performance. Still, the other studies reviewed above suggest a positive relationship between fixation times and learning outcomes. In addition, attention devoted to novel words during reading can depend on their transparency and on contextual support. While we might expect glosses to be an important source of supporting evidence for novel words, none of the reported eye-movement studies included the impact of glosses on reading behaviors or learning outcomes. In the study reported in the following text, we extend previous reading time studies to include time spent fixating on both the novel words and the glosses, allowing us to examine the effect of attention to glosses on reading behavior and on learning outcomes. This may, for example, help to (re-)interpret the advantage (if any) of multimodal over single-mode glosses in terms of quantity rather than quality of processing.
THE CURRENT STUDY
We have highlighted two research strands concerning vocabulary acquisition during reading. One shows the impact of gloss types on learning outcomes, particularly meaning retention. The other uses eye movements to demonstrate a positive relationship between attention paid to novel words and memory for these words, in situations in which readers are usually left to their own devices to discover their meanings. To our knowledge, there is no published research that examines whether the positive relationship between fixation time and uptake in memory helps to account for the superiority (if any) of multimodal glosses over single-mode ones for word learning.
In the current study, learners were exposed to novel words (pseudowords) in a story context. The pseudowords were accompanied by textual, pictorial, or multimodal (text and picture) marginal glosses. While our learners read the story, their eye movements were recorded, providing a record of attention to pseudowords and glosses. Learning was measured by posttests for both pseudoword meaning and form.
Across the story, readers were exposed to three instances of the pseudowords (along with one in the gloss). This adds to the ecological validity of including glosses because these are more likely to be provided for important novel words, which are in turn more likely to recur in a text.
RESEARCH QUESTIONS
We address the following general questions:
• Do multimodal marginal glosses (with both textual and pictorial clarifications of word meaning) help L2 learners achieve better scores than single-mode marginal glosses (with only a textual or pictorial clarification) on postreading tests concerning word form as well as word meaning?
• How much attention (as measured by eye-tracking) is devoted to instances of the novel words and to their glosses, and how is this attention affected by the gloss type?
• To what extent does the amount of attention paid to instances of the novel words and to their glosses predict performance in postreading tests of word form and meaning?
PARTICIPANTS
Our readers were 52 adult high-intermediate ESL learners (30 females, 22 males) enrolled in a general English proficiency program at Victoria University of Wellington.Footnote 1 They volunteered after reading an information sheet that explained the study was about reading in a second language. The precise aim of the study was not specified. The study had approval from the University’s Human Ethics Committee (approval #20143). Learners were randomly assigned to one of three gloss conditions (see following text) and received a NZ$20 supermarket voucher in return for their time.
MATERIALS AND PROCEDURE
The reading text was based on a local news story, was approximately 900 words long, and incorporated six pseudowords. While we recognize that a total of six target items represents a rather small number of experimental tokens, we were keen to ensure that the text remained intelligible and that our readers would read with a primary focus on the story’s content without being distracted by excessive information in the margin. Apart from the six pseudowords, two real English nouns were glossed to reduce the likelihood of participants becoming suspicious of the nature of the target pseudowords. The proportion of eight glossed words in a 900-word text is similar to a recent study by Khezrlou et al. (Reference Khezrlou, Ellis and Sadeghi2017).
Each pseudoword occurred three times in the text body: twice on the same page as the gloss and once on the following page. On its first occurrence in the story, each pseudoword was presented in boldface—typographic enhancement being a typical means in glossing interventions to indicate that an annotation about the word is available—and was accompanied in the right-hand margin by a gloss with one of the following forms:
(1) Pseudoword (bolded) followed by a textual definition (“text only”).
(2) Pseudoword (bolded) accompanied by a picture above it (“picture only”).
(3) Pseudoword (bolded) followed by a textual definition and accompanied by a picture above it (“multimodal”).
Three versions of the text were created, one with each type of gloss. Fifteen participants completed the text-only condition, 19 the picture-only condition, and 18 the multimodal condition.
The second occurrence of each pseudoword in the body of the text was on the same page as the first, an average of 48 words/6 lines later. The third instance was on the following page. The target word was the first content word in the textual definitions. Appendix A gives examples of the three conditions for the pseudoword perchant.
The forms of the glossed pseudowords (panipline, perchant, hangles, dasters, bandilon, and stavener) were borrowed or adapted from Godfroid et al. (Reference Godfroid, Boers and Housen2013) and Godfroid and Schmidtke (Reference Godfroid, Schmidtke, Bergsleithner, Frota and Yoshioka2013). Phonological and orthographic plausibility in English was verified by four native speakers. All pseudowords replaced words with a concrete meaning to enable easy pictorial elucidation. The pictures were color photographs selected from freely available Internet materials. To ascertain that the pictures were unambiguous, nine PhD students read the text with picture-only glosses and subsequently named or described the referent they thought each picture represented. No evidence of picture ambiguity emerged.
The text was distributed over eight screens, with each introducing a new glossed word. Participants pressed the space bar to move to the next screen when ready. They could not return to previous screens. Participants were told that a quiz about the text would follow, but not that it would involve vocabulary. To a degree, then, the current study examines incidental vocabulary learning, although it should be noted that our reading conditions are different from incidental learning conditions in many previous studies because the presence of glosses potentially makes the form more salient, allowing it to attract more attention than it might in other paradigms. The quiz (henceforth posttest) was administered using E-Prime (Psychology Software Tools, 2012) immediately after the reading activity and comprised two parts. In the first part, participants were asked five multiple-choice content questions (each with one correct option and three foils) and eight word-recall questions. Each question was presented on a separate screen. The content questions concerned text passages that did not require comprehension of the pseudowords. The word-recall questions concerned the glossed words: six pseudowords and two existing words. For each word, three recall prompts were presented together on the screen: (a) the sentence in which the target word was first encountered in the text, with a gap instead of the word; (b) the definition from the textual gloss; and (c) the picture from the pictorial gloss. Participants were asked to type as much as they could remember of the form of the word. An example of the recall test for word form is provided in Appendix B.
The second part of the posttest was a matching task. Each pseudoword was presented on a separate screen and participants were asked to identify its corresponding meaning from among 11 options. These consisted of the pictures and definitions (as given in the textual glosses) of the six pseudowords and two real words glossed in the text and of three additional real words that occurred in the text. An example of the matching task used to test meaning retention is provided in Appendix C.
The prompts used in both tests were the same for all gloss conditions. It could be argued that confronting some participants with pictorial representations that they had not seen during their reading activity may have created an additional processing load at the test stage. Likewise, giving participants textual definitions that they had not previously seen may have induced extra processing. Note that the noncongruency between gloss condition and test prompts that occurred in the context of previous studies (e.g., Kost et al., Reference Kost, Foss and Lenzini1999) was where participants in some conditions were required to rely exclusively on test prompts they had not seen during reading. Our primary rationale for using identical tests for each condition, including a combination of prompts, was that we could be sure that they at least included the meaning representation available during reading. In addition, using a tailor-made test for each condition would arguably have introduced a confounding factor, where different outcomes might have been attributed to the test condition rather than to the reading condition. Finally, no time pressure was imposed in the posttests, and so, even though the presence of new stimuli probably invited extra processing in two of the conditions, this is unlikely to have had a detrimental effect on the accuracy of responses due to an excess of information.Footnote 2 The reading and testing procedures together took about 20 minutes.
Participants were randomly assigned to one of the three conditions. They came from five language classes at different levels, to which they had been assigned based on an English proficiency test. Because we did not have human ethics approval to access participants’ proficiency test scores, we used their class level as a proxy for proficiency. There was no significant difference in the class level of participants in the three conditions (by chi-square test, p = .98). In addition, there was no significant difference between groups in their scores on the general comprehension questions about text content (Kruskal-Wallis χFootnote 2(2) = .24; p = .89).
APPARATUS
Participants were tested individually. A forehead and chin rest maintained a viewing distance of 57 cm to a 21-inch computer monitor, which presented pages of text at a resolution of 1024 × 768 pixels and a refresh rate of 60 Hz. At this distance and resolution, text presentation averaged three letters per degree of visual angle. Eye movements and positions were measured with an EyeLink 1000 Tower Mount Head Supported system (SR Research Ltd., 2011). A video-based infrared camera measured corneal reflection and the dark pupil of the right eye using an infrared reflective mirror. Positions were sampled at 1,000 Hz, and at a spatial resolution of 0.01 degrees of visual angle. Experiment presentation was controlled using SR Research Experiment Builder Version 1.10.165 (2011).
Fixation data were tabulated for several predefined interest areas, including the three presentations of each pseudoword in the text, the pseudoword in the marginal gloss, the picture in the gloss (for conditions including a picture), and the textual gloss (for conditions that included this). Fixations were extracted using EyeLink Data Viewer Version 1.11.9000 (2007). A research assistant blind to our hypotheses conducted drift correction using Data Viewer’s semi-automatic algorithm. The procedure involved placing the first fixation on each line of text manually, followed by automatic correction of subsequent fixations on the line. After drift correction, fixations were cleaned in a four-step process by which brief fixations within a radius of 0.5 degrees were merged. After merging, fixations of less than 140 ms or greater than 800 ms were deleted. This procedure resulted in a loss of 8.46 percent of fixations, which is within typical levels and, importantly, did not differ by gloss condition.
ATTENTION MEASURES
We examined three eye-tracking measures: first fixation duration (FFD), total fixation counts (TFC), and total reading time (TRT). These serve as proxies for the amount of attention paid by participants to each of the three occurrences of the pseudowords in the body of the text and to the marginal glosses (and their components: the pseudoword, its textual definition, or its picture). In terms of the relationships between eye-tracking measures and cognitive processing discussed by Johnson and Mayer (Reference Johnson and Mayer2012: 181–182), we are considering measures of attentional focus on pseudowords and glosses, rather than measures of integration of pseudowords and glosses, which these authors argue are best measured by considering transitions (i.e., saccades) between the text and the glosses (see also Mason, Pluchino, & Tornatora, Reference Mason, Pluchino and Tornatora2016, who focus in particular on transitions between text and illustration during second pass reading).
An additional measure is whether any attention is paid to the interest area, or whether instead it appears to be ignored. Reading studies have shown that if a word is predictable in its context, then it is frequently skipped (Rayner, Slattery, Drieghe, & Liversedge, Reference Rayner, Slattery, Drieghe and Liversedge2011), with as many as a third of all words skipped in initial reading (Rayner, Reference Rayner1998, Reference Rayner2009). In our study, fixation likelihoods for the pseudowords were at or close to ceiling. This is not unexpected—participants were nonnative speakers who mostly read slowly through the text, and the pseudowords were both unknown and appeared in the types of sentence position expected of content words that are highly likely to be fixated (Rayner, Reference Rayner1998: 375 reports that native English speakers fixate content words approximately 85 percent of the time and function words 35 percent of the time).
Nevertheless, the fact that words (and glosses) can be skipped is problematic for the analysis of FFDs and TRTs, and in particular for whether averages of these measures should be based on just those cases in which there are measurable fixations or should also include zero values. Rayner (1998: 376–378) points out that words can be processed during reading even when they are not fixated because some features such as overall length and shape will become available to the reader (through parafoveal preview) during fixation on earlier words. He discusses several solutions to the problem of what to do with nonfixated words. Many of these solutions are less appropriate for our study, where we are interested not just in fixations on individual words during reading, but also in reading behaviors over larger areas of interest such as the entire marginal gloss or the picture in a gloss. In this context it is important to mention that the glosses were more frequently ignored, that is, not fixated at all, than the pseudowords. This was particularly striking in the case of the text-only glosses, which were ignored 20 percent of the time. This compares to 11.4 percent and 9.3 percent for the picture-only and multimodal glosses, respectively. If text-only glosses were ignored so much more often than the other two gloss types, this inevitably has implications for whether we should exclude missing fixations from our comparisons of average fixation counts and average TRTs. We chose to include zero fixations, however, because we are interested in how (if at all) different gloss types influence reading behavior. If some gloss types are more likely to be ignored, then that is a pedagogically pertinent finding.
Our analysis of TRTs, as a measure of total attention given to an item, includes two components: skip rate (i.e., the probability that the item receives no fixation) and the total time spent fixating the item, conditional upon skip rate. For the analysis of FFDs, however, we follow Pollatsek et al. (Reference Pollatsek, Reichle, Rayner, Hyönã, Radach and Deubel2003: 371) and Murray (Reference Murray and Underwood1998, 190n4) in treating any skipped instances of the item in question as missing values. Therefore, the average FFDs reported in the following text should be interpreted as indicating the amount of attention given to an item on first seeing it, given that it receives any attention at all.
DATA ANALYSIS
In the form recall test, participants seldom supplied a fully correct target word. Recall attempts usually resulted in partially correct responses (e.g., just the first letter or syllable). Two blind judges independently scored each response on an 11-point scale from 0.0 (no form recall attempt or a completely incorrect response) to 1.0 (completely correct recall). Intermediate scores included 0.1 when only the first letter was given, 0.3 for the incomplete response pan (for panipline), and 0.8 for the nearly accurate response banlion (for bandilon). Because interrater agreement was high (r = .98), we used the average score of the two judges in our analysis. The responses on the meaning recognition test were scored in a binary fashion (i.e., either correct or wrong).
The following results section presents analyses of how readers directed their attention (with respect to the pseudowords and the glosses); of the overall impact on form recall and meaning recognition of different gloss types; and of the relationship of attention to form recall and meaning recognition. For most of the statistical analyses of the attention measures and the relationship of these to form and meaning retention, we ran mixed-effects models using lmer from the lme4 package (Bates, Maechler, Bolker, & Walker, Reference Bates, Kliegl, Vasishth and Baayen2015) in R (R Core Team, 2014). Linear models were used in all cases except for when meaning recognition (a binary variable) was the dependent variable, in which case logistic models were used. When the dependent variable was TFCs, the linear model assumed the Poisson distribution that is appropriate for count data. Attention measures that involved durations (FFD and TRT) were log-transformed prior to statistical analysis to provide a better fit to a normal distribution. For TRTs we used a zero-inflation approach to mixed-effects modeling using glmmTMB (Magnusson et al., Reference Magnusson, Skaug, Nielsen, Berg, Kristensen and Maechler2017). This permitted a two-part analysis: a logistic model predicted the likelihood of skipping the item in question and a linear model predicted reading time (based on log values), conditional on skip rate.
Barr et al. (Reference Barr, Levy, Scheepers and Tily2013) recommend fitting a maximal random effects structure, with appropriate by-participant and by-items slopes for fixed effects. However, in our data this typically resulted in overfitting and model nonconvergence, which is not surprising given the small number of items (pseudowords). For each model reported as follows we therefore obtained an optimal random effects structure following the procedure outlined by Bates et al. (2015). The significance level of fixed-effect predictors was assessed using model comparison using likelihood ratio tests with the mixed function in the afex library (Singmann, Bolker, & Westfall, Reference Singmann, Bolker and Westfall2015). Note that mixed uses sum contrast coding, comparing each level of a factor to the average of the other levels. Post hoc comparisons, with Tukey p-value correction for multiple contrasts, were performed on model outputs through least-squares means using lsmeans (Lenth, Reference Lenth2016). Further details of statistical model design, including the optimal random effects structures, are given in relevant parts of the results section.
RESULTS
ATTENTION
In this section we present analyses of the attention paid to the pseudowords and their glosses during participants’ reading of the text. Because the glosses differed by condition, separate analyses involved attention paid to the pseudowords and to the glosses. Recall that each pseudoword appeared three times in the text and once in the gloss. FFDs, TFCs, and TRTs were the dependent variables in separate mixed-effects regression analyses. Fixed effects were Instance (the three instances of the pseudoword in the text and the one instance in the gloss), Condition (multimodal, picture only, text only), the interaction of these factors, as well as pseudoword Length (in characters) for analyses involving pseudowords (because word length is widely recognized to have an effect on fixations, e.g., Rayner, Reference Rayner1998, Reference Rayner2009), and the Class level of the participant (as a proxy for proficiency). The factor Instance was treated as categorical because there was no fixed order in which the four occurrences of the pseudoword had to be inspected, save that the third instance in the text was always seen last because it occurred on a following screen page. A descriptive summary of results is provided in Table 1.
Eye-tracking measures (FFD = mean FFD, in milliseconds; Count = mean fixation count; Skip = skip rate in percent; TRT = mean TRT, in milliseconds; TRT′ = mean TRT, conditional upon the region not being skipped. The first four lines in each condition refer to instances of the pseudowords)
First Fixation Durations
The optimal random effects structure for the analysis of FFDs on the pseudowords included just intercepts for participants and pseudowords. The significant effects returned by model comparison were Instance (χFootnote 2(3) = 27.15, p < .0001) and the interaction of Instance and Condition (χFootnote 2(6) = 17.09, p < .01). The Instance effect reflects significantly shorter FFDs on the pseudoword in the gloss—despite it being in boldface—than on each of the instances in the text (p < .02 in least-squares means comparisons). FFDs for the instances in the text did not differ from one another. As can be seen from Figure 1, the interaction reflects differences in the patterns of FFDs across the four instances in the different conditions. Post hoc pairwise comparisons showed no significant differences between any of the instances in the picture-only condition. In the multimodal condition, FFDs on the first instance were significantly greater than those on the instance in the gloss (t = 4.45, p < .001) and on the third instance (t = 2.65, p < .05), and longer on the second instance than on the instance in the gloss (t = 3.20, p < .01). In the text-only condition, both the first instance and the instance in the gloss had shorter fixations than the second instance (t = -2.79, p < .05 and t = -3.76, p < .01), and the instance in the gloss had shorter fixations than the third instance (t = -2.68, p < .05). There is thus little evidence here that typographic enhancement (boldface) of the first instance of the word in the text and of its iteration in the gloss triggered consistently longer FFDs. This is surprising, given that typographic enhancement had an attention-directing effect in other eye-tracking studies, such as Winke (Reference Winke2013) and Choi (Reference Choi2017)—but see below for fixation count data that accords better with expectations.
Two-way interaction of Instance and Condition for FFDs on the pseudowords. (Means and standard errors. Values shown are those predicted by the mixed-effects model, back-transformed from log values to millisecond values for transparency.)
FFDs in the definition area were compared in the multimodal and text-only conditions because there was no definition in the picture-only condition. Similarly, the picture area could only be compared in the multimodal and picture-only conditions. In both analyses, the mixed-effects models tested for Condition and Class as fixed effects, and the random effects structure included intercepts for participants and pseudowords, and random slopes for Condition across pseudowords. The only significant effect for the definition area was Class, with longer first fixations for participants in the more advanced classes (χFootnote 2(1) = 4.21, p < .05). Neither Condition nor Class showed significant effects for first fixations in the picture area.
Fixation Counts
We turn now to the analysis of TFCs, starting with the pseudowords. The optimal random effects structure included intercepts for both participants and pseudowords, as well as participant and pseudoword slopes for Instance. The analysis returned significant effects of Instance (χFootnote 2(3) = 18. 374, p < .001) and Length (longer pseudowords had more fixations: χFootnote 2(1) = 5.97, p < .05). The model predicted more fixations on the first instance of the pseudoword than on the others (each of which was significantly different from the first instance by least-squares means at p < .01 and did not differ from one another). Condition had no effect on the number of fixations on the pseudowords.
For TFCs in the definition and picture areas, the analyses again tested for Condition and Class, with random intercepts for participants and pseudowords and random slopes for Condition across pseudowords. Neither fixed effect was significant in the analysis for the definition area. For the picture area there was a significant effect of Condition (χFootnote 2(1) = 5.39, p < .05), with more fixations in the picture area in the picture-only condition than in the multimodal condition. Class was not significant.
A comparison of the aggregated TFCs in all areas of interest concerning the pseudowords, that is, their instances in the text plus their associated gloss components, shows that numerically the multimodal (17.5) and text-only gloss conditions (16.8) had more fixations than the picture-only gloss condition (13.8), but this difference was not significant (p = .15).
Total Reading Time
The optimal random effects structure for the zero-inflation analysis of TRT on pseudowords included just random intercepts for both participants and pseudowords. There were significant overall effects of Instance (χFootnote 2(6) = 213.00, p < .0001) and Length (χFootnote 2(2) = 6.63, p < .05). Consideration of the two components of the optimal model shows that the logistic analysis of skip rate produced a significant effect of Instance, but not of Length, while the linear analysis of TRT conditional on skip rate was significant for both Instance and Length. The Length effect was that longer pseudowords had greater TRTs. To assess significance of the differences between scores for each instance, models were run with different instances as the baseline value against which the other instances were compared (by z-test). There were significantly fewer skips of the first instance of the pseudoword than any other instance and significantly more skips of the pseudoword in the gloss than for any of the other instances (p < .001 in each case). Skips for the second and third instance did not differ from one another. TRT was significantly greater for the first instance than for any of the others, and significantly shorter for the second instance than for any of the others (p < .01 in each case). TRTs for the third instance and for the instance in the gloss did not differ from one another.
Analysis of the entire definition area (excluding the picture-only condition, which did not have a textual definition) included Condition and Class as fixed effects. The optimal random effects structure included only intercepts for participants and pseudowords. There were no significant effects. The analysis for the picture area (excluding the text-only condition) similarly included Condition and Class as fixed effects, but this time the optimal random effects structure included by-pseudoword slopes for Condition as well as intercepts for both participants and pseudowords. There were no significant effects in the logistic part of the model, that is, skip rates did not differ by Condition. TRTs did, however, differ significantly by Condition (z = 2.5, p < .05). The average total time spent looking at the picture was longer in the picture-only condition than in the multimodal condition. The fact that readers spent more time inspecting the picture in the former condition is not unexpected as the picture was the only information available to them in the gloss area to figure out the word’s meaning.
We also ran regression models for overall fixation counts and TRT on the set of four instances of each pseudoword pooled together, with Condition, Length, and Class as fixed-effect predictors, along with random intercepts for participants and pseudowords and by-pseudoword slopes for Condition. The analysis of summed fixation counts returned no significant effects. When all four instances are pooled in this way, skip rates are virtually nil, meaning that a zero-inflation analysis of TRT was not possible. The analysis of TRT is therefore of the time spent reading the pseudowords, summed over the four instances, without consideration of skip rate. The only significant effect was of pseudoword length (χFootnote 2(1) = 5.47, p < .05), with longer reading times for the longer pseudowords.
Finally, we also added up the TRTs per pseudoword and its associated gloss components. Altogether, the words and their glosses attracted the greatest average TRT in the multimodal condition (4,659 ms), followed by the text-only gloss condition (4,502 ms). The mean value was markedly shorter in the picture-only gloss condition (3,751 ms). This is perhaps not surprising, as there was no textual definition for the participants to “take in,” and it stands to reason that glancing at an elucidating illustration takes less time than reading a definition. It is worth noting, however, that the difference in cumulative TRTs between the three gloss conditions nevertheless falls short of significance (p = .20).
Summary
Altogether, the attentional measures lend only modest support to the thesis that gloss types have different effects on fixation behavior. Less time tended to be spent on the definition when a picture was also available in the gloss—which could be interpreted either as competition for attention or as a reflection that the picture facilitated fast processing of definitional content—but this difference did not reach statistical significance. Conversely, when no definition was available in the gloss, the picture received more attention, and this difference did reach significance.
As to the three instances of each target word in the text, the first instance attracted most fixations and longer TRTs. This was to be expected, not only because it was the first encounter but also because it was typographically enhanced. Interestingly, the third instance, which appeared on the next screen and without a clarifying gloss, also tended to attract comparatively more attention. However, the gloss conditions did not appear to differently affect the amount of attention given by the readers to the three pseudoword instances in the body of the text.
POSTREADING WORD-FORM RECALL AND MEANING RECOGNITION
Mean scores in the form recall and meaning recognition tests are shown in Table 2. In the form-recall test the picture-only condition yielded the best results, although the effect of Condition was not significant (χFootnote 2(2) = 3.53, p = .17). Given the overall lack of indications from the online reading measures that the three gloss conditions directed attention to the pseudowords to different degrees, this is unsurprising. We did find a significant effect of Class (χFootnote 2(1) = 4.86, p < .05), showing that the more advanced learners were more accurate in their recall of the form of the pseudowords.
Postreading word-form recall and meaning recognition by gloss condition (max = 1 in both cases)
The scores in the meaning recognition test did differ significantly by Condition (χFootnote 2(2) = 6.42, p < .05). Average scores in the text-only condition were significantly lower than those in the picture-only condition. Scores in the multimodal condition lay between these two but were not significantly different from either. In other words, there was a general benefit for meaning retention of having the picture present in the gloss, but this is less marked for the multimodal condition than for the picture-only condition. Class was not significant as a predictor of meaning recognition.
If fixations and reading times are predictive of learning—as has been attested in several studies (see “Background” section), then it is intriguing that the picture-only glosses appeared to foster the best retention of form-meaning connections, given that we see no evidence that pictures enhance uptake by affecting the attention paid to words or the glosses. Indeed, the aggregated TFCs and TRTs were numerically (though not significantly) lowest in the picture-only condition.
EFFECTS OF ATTENTION PAID TO THE PSEUDOWORDS
To determine whether the amount of attention paid to the pseudowords in the text influenced their uptake, analyses were carried out in which scores in the tests for form recall and meaning recognition were predicted by each of the three measures derived from our eye-tracking data. Separate series of mixed-effects regression models were run with form recall and meaning recognition as dependent variables. In each series, separate models included each of the reading measures (FFD, TFC, or TRT) as a fixed-effect predictor. Pseudoword Length and the Class level of the participants (as a proxy for proficiency) were also included as predictors. Initial models considered the interaction of the reading measure with the Instance of the pseudoword (first, second, or third in the text, or the one in the gloss). The presence of such an interaction would indicate a differential effect on the dependent variable (form recall or meaning recognition) of attention paid to each instance of the pseudoword. When there was such an interaction, it was further explored through separate models for each Instance. Our models also included Condition (text only, picture only, and multimodal) and its interaction with the reading measure, as well as the three-way interaction of Condition, Instance, and the reading measure (to assess the possible impact of differences in the attention paid to the various instances of the pseudoword that might result from differences between the gloss conditions).
Form Recall
The optimal random effects structure for the model testing the effects on form recall of FFD included random intercepts for participant and pseudoword and by-pseudoword slopes for Condition. The model returned a significant effect of Instance (χFootnote 2(3) = 8.36, p < .05) and an interaction of this with FFD (χFootnote 2(3) = 8.42, p < .05). The only other significant effect was of Class—as reported earlier, participants in the more advanced classes showed better form recall (χFootnote 2(1) = 4.79, p < .05). The interaction of Instance with FFD is shown in Figure 2. Further exploration of this interaction considered the effect on form recall of FFD for each instance separately, with Condition, Length, and Class included as before. For each of the three instances of the pseudowords in the text the only significant effect was the participants’ Class level (in each case p < .05, as in the overall analysis). For the instance of the pseudoword in the gloss, there was additionally a significant positive effect of FFD (χFootnote 2(1) = 8.09, p < .01).
Two-way interaction of Instance of the pseudoword and FFD in predicting form recall score. (Values shown are those predicted by the mixed-effects model. Shaded ribbons show standard errors.)
For our model including the effects of fixation count in predicting form recall, the optimal random effects structure included only intercepts for participants and pseudowords. The only significant effects were Class (better recall for higher classes, χFootnote 2(1) = 5.37, p < .05) and TFC (χFootnote 2(1) = 6.62, p < .05). A larger number of fixations on the pseudowords had an overall facilitative effect on form recall, and this was not affected by gloss Condition nor by which Instance of the pseudoword was fixated.
The model predicting the effect of TRT included random intercepts for participants and pseudowords and by-pseudoword slopes for Condition. The only significant predictor was participants’ Class level (χFootnote 2(1) = 5.33, p < .05). The TRT spent on each pseudoword had no effect on the recall of form, neither as a simple effect nor in interaction with either Condition or Instance.
The impact of attention paid to the pseudowords across the entire reading passage was assessed by summing TFCs and TRTs for all four instances of each pseudoword. Summed TFCs were used as a predictor in a model that also included Condition (and its interaction with TFC) and Class, along with an optimal random effects structure of participant and pseudoword intercepts and by-pseudoword slopes for Condition. The model returned Class (χFootnote 2(1) = 7.08, p < .01) and TFC (χFootnote 2(1) = 12.50, p < .001) as significant predictors of form recall, with no other effects. A parallel process was followed for summed TRTs, except that by-pseudoword slopes for Condition had to be dropped before the model would converge. Summed TRT significantly predicted form recall (χFootnote 2(1) = 4.60, p < .05). There were no other significant effects.
These results show that form recall increases with the total number of fixations on the pseudowords, and the total time spent looking at them. These appear to be general findings not specific to individual instances of pseudowords. However, we also found that the impact of initial attention paid to pseudowords (as measured by FFD) was limited to the instance in the gloss. None of these effects were influenced by gloss Condition.
Meaning Recognition
As with form recall, the analysis involved separate models in which the scores for meaning recognition were predicted by each of the three different reading measures in interaction with Instance, as well as by Condition, the interaction of Condition with the reading measure, and the three-way interaction of Condition, Instance, and the reading measure. The Length of the pseudoword and the Class level of the participant were also included as fixed effects. The random effects structure for the analyses of FFD and of TRT included random intercepts only. For TFCs, it also included by-pseudoword slopes for Condition. None of the models returned any significant effects.
While there was no impact on meaning recognition of attention paid to individual instances of the pseudowords, further models including summed TFCs and TRTs as predictors showed a different picture. The first model showed a significant effect of TFC summed across all four instances (χFootnote 2(1) = 7.62, p < .01), as well as a significant effect of the participants’ Class level (higher proficiency participants showed better meaning recognition; χFootnote 2(1) = 3.86, p < .05). The second model showed a significant effect of summed TRT (χFootnote 2(1) = 4.60, p < .05). In other words, a greater cumulative exposure to the pseudowords bears a positive relationship to meaning recognition.
EFFECTS OF ATTENTION PAID TO THE INFORMATION PROVIDED BY THE GLOSSES
We turn now to the effects on form recall and meaning recognition of attention paid to areas of the marginal gloss. Analyses of reading measures for the picture involve comparisons of the picture-only and multimodal conditions, while those for the definition involve the text-only and multimodal conditions. The optimal random effects structure in both models included by-pseudoword slopes for Condition as well as intercepts for participants and pseudowords. Neither analysis returned any significant effects. It appears that the amount of attention paid to the picture and to the definition in the gloss area had no effect on either the recall of the form or recognition of the meaning of the pseudowords.
DISCUSSION
The first general question we set out to answer was whether multimodal glosses are superior to single-mode glosses as regards vocabulary uptake. Our posttest data yield mixed findings. According to the descriptive statistics (Table 2), multimodal glosses appeared to promote better uptake of both form and meaning than text-only glosses, but these differences fell short of statistical significance. Instead, it was the second single-mode gloss type, that is, the picture-only gloss, that appeared to promote the best uptake of both form and meaning in the present study, although scores in this condition were significantly different only from those in the text-only condition, and only for meaning recognition. In sum, the test results yield no compelling evidence in favor of multimodal glosses but are favorable of picture-only glosses—at least in the case of word meanings that are depicted in an unambiguous way. These results differ from several previous studies (see the “Background” section) that claimed superiority of multimodal glosses over both text-only and picture-only glosses.
The second research question was whether the gloss types influence the amount of attention given to the target words or the glosses. Altogether, we found very little evidence that the three gloss conditions affected the distribution of attention. The one significant difference that emerged was the increased time given by participants to the picture in the glosses when this was the only elucidation presented to them, in comparison to the multimodal glosses where participants appeared to divide their attention between the picture and the textual clarification. The total, cumulative amount of time that participants fixated the words and their glosses was greatest in the multimodal condition but did not differ significantly among the three conditions.
Regardless of gloss condition, the amount of attention decreased quite dramatically from first to subsequent instances of the target words in the text. Pellicer-Sánchez (Reference Pellicer-Sánchez2016), Mohamed (Reference Mohamed2018), and Elgort et al. (Reference Elgort, Brysbaert, Stevens and Van Assche2017) also found decreasing reading times on repeated occurrences as the novel words gradually became more familiar, but the sharp decline observed in the present experiment is nonetheless striking. It is likely that the typographic enhancement and the realization that the word was glossed prompted longer processing of the first instance of the word. Also, given that the gloss clarified the word when first encountered, subsequent instances of the word may have prompted much less processing because it was no longer puzzling.
Moving on to the question of whether any of the online reading measures predicted vocabulary uptake, the findings corroborate previous studies in that total attention given to the target words was positively associated with test performance. And yet, posttest performance appeared the best under the picture-only condition, even though the total amount of time spent on the targets and their glosses was not greater in that condition than it was in the other two conditions. This disconnect between eye-tracking data and posttest scores suggests that pictorial glosses can lend a mnemonic advantage, but not because they engender protracted visual processing of either the gloss or the pseudowords. Note that the uptake of information from visual scenes is very rapid compared to reading—meaning can be extracted from complex visual scenes in as little as 100 ms (Biederman, Rabonowitz, Glass, & Stacy, Reference Biederman, Rabonowitz, Glass and Stacy1974), that is, within a single fixation. This advantage may be particularly acute for second-language learners, for whom the meaning of the picture should be more transparent than the textual definition.
The question remains, though, why the combination of pictorial and textual representation of word meaning (in the multimodal gloss condition) was not also significantly more effective than the text-only gloss condition. After all, participants in the multimodal gloss condition also looked at the pictures, according to the eye-tracking data (although TFCs were lower and TRTs shorter than when only a picture was available). One possible explanation is that the presence of text (which might have been relatively hard for them to understand) may have interfered with the extraction of meaning from the picture. Another is that the interpretation of the picture without a textual clarification took slightly more effort (hence perhaps the slightly longer time spent on the pictures), and this generated stronger memories (as would be predicted under models such as levels of processing; e.g., Craik & Lockhart, Reference Craik and Lockhart1972).
CONCLUSION AND LIMITATIONS
Our study aimed to evaluate the effects of three different types of marginal glosses (text only, picture only, and text and picture) on L2 learners’ uptake of new words from reading. The novelty of the study was to examine not only learning outcomes (through posttests) but also online reading processes (through eye-tracking measures). We examined whether gloss type influences reading behaviors and, if so, whether these reading behaviors (especially the amount of attention given to target words and to the information contained in glosses) help to predict learning outcomes.
Two postreading tests were administered: a productive-knowledge test, where participants were prompted to recall the form of the target words, and a receptive-knowledge test, where they matched target words with their meaning. No difference emerged between scores on the first test under the three gloss conditions. The second test, however, revealed an advantage for glosses that included a picture, especially for the picture-only glosses.
In general, posttest performance was positively associated with the amount of attention participants gave to the target words and their glosses during reading, corroborating previous studies where eye-tracking data were predictive of postreading test scores. The eye-tracking measures in the present study revealed only minimal differences in online processing of the target words and their glosses across the three gloss conditions, although one striking finding here was that text-only glosses tended to be skipped (i.e., ignored) far more often than glosses containing a picture. Another finding was that the target word reiterated in the gloss attracts very little attention, even if it is typographically enhanced (using boldface, in this experiment). This suggests that when readers realize a word comes with a gloss and make the effort to consult it, they will promptly turn their attention to the clarification of the word rather than to the instance of the word. Interestingly, though, the length of their first fixation on the word in the gloss does appear to positively influence their retention of the word’s form.
Altogether, the eye-tracking data provided little to explain why posttest performance tended to be the best after learners had read the text accompanied by picture-only glosses. This suggests that more facets of online processing play a part in establishing memories for words than meet the eye(-tracker). These might include contextual factors, although in the current case the text contexts at least were the same for each gloss condition. It is of course possible that the eye-tracking measures we opted to use here were insufficient to pick up relevant differences in participants’ allocation of attentional resources. Deploying a larger arsenal of measures, including transitions (saccades) between areas of interest, might reveal additional differences.
Several other limitations to this study must be acknowledged. One concerns sample sizes, not only regarding the number of participants but also with regard to the small number of target words. The fact that only two tests were used to measure learning outcomes is another limitation because these may not have been sensitive enough to pick up differential learning gains at a subtle level of word-knowledge development. Finally, although the purpose of this study was not to measure the effects of glossing per se but rather to compare the effects of different gloss types, data from a control condition without any glossing could have been informative to evaluate the extent to which glossing as such influences reading behavior. Despite these limitations, we hope this study will stimulate further investigations of text manipulations (such as glossing) intended to foster vocabulary uptake, and as well as where investigations online reading measures shed light on offline learning outcomes.
APPENDIX A
EXAMPLES OF TEXT AND GLOSSES FOR THE PSEUDOWORD PERCHANT
In order, these are text-only, picture-only, and multimodal gloss conditions.
APPENDIX B
EXAMPLE OF THE POSTTEST USED TO MEASURE RECALL OF WORD FORM. SEE TEXT FOR DETAILS.
APPENDIX C
EXAMPLE OF THE MATCHING TASK USED TO TEST MEANING RETENTION. SEE TEXT FOR DETAILS.
