HOW DO DIFFERENT FORMS OF GLOSSING CONTRIBUTE TO L2 VOCABULARY LEARNING FROM READING? A META-REGRESSION ANALYSIS
Research has demonstrated that L2 students can incidentally learn unknown words from reading (e.g., Day, Omura, & Hiramatsu, Reference Day, Omura and Hiramatsu1992; Horst, Cobb, & Meara, Reference Horst, Cobb and Meara1998; Pigada & Schmitt, Reference Pigada and Schmitt2006; Webb & Chang, Reference Webb and Chang2015). However, studies tend to show that learning gains from reading are relatively small. This is mainly because inferring the meanings of unknown words in context is challenging (e.g., Nassaji, Reference Nassaji2003). For example, to successfully guess a word’s meaning, learners need to know the majority of the words in a text to understand the context (Hu & Nation, Reference Hu and Nation2000; Laufer, Reference Laufer1992; Schmitt, Jiang, & Grabe, Reference Schmitt, Jiang and Grabe2011), and in turn provide the greatest potential of inferring the unknown word (Liu & Nation, Reference Liu and Nation1985). One solution to this issue is to provide glosses for unknown words (Webb & Nation, Reference Webb and Nation2008). Research has demonstrated that glossing or annotating unfamiliar words not only promotes L2 reading comprehension but also increases learning of the glossed words (e.g., Hulstijn, Hollander, & Greidanus, Reference Hulstijn, Hollander and Greidanus1996; Jacobs, Dufon, & Hong, Reference Jacobs, Dufon and Hong1994; Watanabe, Reference Watanabe1997).
Many studies have compared different forms of glossing to determine how glosses can maximize vocabulary learning from reading. Some studies have compared glossing types such as marginal glosses, multiple-choice glosses, in-text glosses, and hyperlinked glosses (Nagata, Reference Nagata1999; Watanabe, Reference Watanabe1997). Other studies have examined which language (L1 or L2) should be used to provide the meanings in glosses (Jacobs et al., Reference Jacobs, Dufon and Hong1994; Ko, Reference Ko2012; Yoshii, Reference Yoshii2006), or the mode (textual, visual, auditory) in which glosses are presented (Kost, Foss, & Lenzini, Reference Kost, Foss and Lenzini1999; Yoshii & Flaitz, Reference Yoshii and Flaitz2002). However, despite a great deal of research on the efficacy of different forms of glossing, there tend to be contradictory conclusions, and there remains a lack of consensus on the relative values of the different approaches to glossing.
One way to deepen our understanding of how glosses should be provided is to conduct a meta-analysis. Individual studies are restricted, for example, by their research foci, participant population, target languages, and materials. In contrast, meta-analysis allows the examination of how the effects of glossing vary in relation to variables such as gloss formats (gloss type, language, mode), characteristics of texts (e.g., learner targeted texts, proportion of glossed words) and learners (e.g., institutional level, proficiency) by synthesizing studies conducted in different contexts. Results from meta-analyses are usually more reliable because they are based on the results of multiple studies. Furthermore, the increased sample sizes through meta-analysis provide a higher statistical power (Hunter & Schmidt, Reference Hunter and Schmidt2004), which enables the reexamination of the value of variables that have been rejected due to low statistical power. The present meta-analysis aims to investigate (a) the overall learning gain from glossed reading, and (b) how glossing effects vary based on format (i.e., type, language, mode) and the characteristics of texts and learners. The current meta-analysis may provide substantial pedagogical implications as to how glosses should be provided for language teachers and material designers.
BACKGROUND
Generally, glossing in the L2 learning context refers to providing L1 translations, L2 synonyms, or short explanations of unfamiliar words in a text (Bowles, Reference Bowles2004; Nation, Reference Nation2013). Many studies have demonstrated that glossed reading leads to greater vocabulary learning than non-glossed reading (e.g., Hulstijn et al., Reference Hulstijn, Hollander and Greidanus1996; Jacobs et al., Reference Jacobs, Dufon and Hong1994; Ko, Reference Ko2012; Watanabe, Reference Watanabe1997).
Glossing has many attractive features. For example, glosses are easier to access than dictionaries, and their presence in a text allows learners to continue on with the reading process with little interruption (Yanguas, Reference Yanguas2009). This may lead learners to check the meanings of words more frequently using glosses than other resources (Hulstijn et al., Reference Hulstijn, Hollander and Greidanus1996). Also, glosses enhance learners’ noticing of unknown target language items (Bowles, Reference Bowles2004; Rott, Reference Rott2005; Yanguas, Reference Yanguas2009), and ensure appropriate form-meaning connections (Nation, Reference Nation2013). This may help readers to learn the meanings of target words efficiently. Furthermore, glossed reading provides opportunities for learners to foster their autonomy by lessening the need to depend on teachers for explanations (Jacobs et al., Reference Jacobs, Dufon and Hong1994; Nation, Reference Nation2013). Researchers’ focus has therefore shifted from whether glosses promote vocabulary learning to how glosses should be provided to maximize their effectiveness (Azari, Reference Azari2012; Mohsen & Balakumar, Reference Mohsen and Balakumar2011; Yoshii, Reference Yoshii2006).
GLOSS TYPE
There are many types of glosses, which can be roughly categorized into noninteractive glosses and interactive glosses. Noninteractive glosses are inserted at a specific place (e.g., margin of a text) and are clearly presented for learners’ use. Noninteractive glosses include marginal (provided in the margin of a text, usually at the right hand or bottom), interlinear (provided between the lines of a text), in-text glosses (provided next to the word in a text), and glossaries (provided at the end of the text or as a separate paper in the form of a list). Paper reading materials are still frequently used, and these noninteractive glosses are adopted as common approaches among language teachers.
Interactive glosses require learners to take action. One example of interactive glosses is hyperlinked glosses, which require learners to click on or put a mouse cursor over a word to see the gloss. Webb and Nation (Reference Webb and Nation2017) argue that hyperlinked glosses are particularly useful because they allow for retrieval of the word meaning. While noninteractive glosses tend to be provided in places that can be easily seen, hyperlinked glosses allow learners to retrieve the meanings of glossed words before opening the glosses. Another example of an interactive gloss type is multiple-choice glosses. Multiple-choice glosses tend to be provided at the margin of texts. Instead of simply providing the corresponding meanings of words, there are several options for a target word meaning. Learners are required to read texts carefully and select the appropriate option that fits the context. Hulstijn (Reference Hulstijn, Arnaud and Béjoint1992) argues that multiple-choice glosses cause deeper processing of glossed words, thus leading to greater vocabulary learning.
Earlier studies have compared the relative effectiveness of different types of glosses. Multiple-choice glosses are one of the most frequently investigated gloss types. Studies comparing multiple-choice glosses and single-translation glosses have reported mixed results. While some studies (Hulstijn, Reference Hulstijn, Arnaud and Béjoint1992; Miyasako, Reference Miyasako2002; Watanabe, Reference Watanabe1997) did not find clear differences between the two types of glosses, others (Nagata, Reference Nagata1999; Rott, Reference Rott2005) found that multiple-choice glosses led to greater vocabulary learning than single-translation glosses. Furthermore, Yoshii (Reference Yoshii2013) found that single-translation glosses led to greater learning than multiple-choice glosses.
In-text glosses are thought to be the least effective type of glossing (Nation, Reference Nation2013; Schmitt, Reference Schmitt2008). Watanabe (Reference Watanabe1997) did not find a clear difference between an in-text gloss condition and a nonglossed condition. However, Cheng and Good (Reference Cheng and Good2009) showed that an in-text gloss condition yielded greater learning than a nonglossed condition. For this reason, it might be too early to conclude that in-text glosses have no effect on learning.
GLOSS LANGUAGE
Whether the meanings of words should be glossed in the L1 (translations) or L2 (definitions or synonyms) has been a long-standing question in vocabulary learning. One advantage of L1 glosses relates to providing short and clear meanings for unknown words regardless of learners’ proficiency levels. Learners are less likely to misinterpret the meanings of glossed words when L1 glosses are presented compared to when words are glossed with L2 definitions. However, L2 glosses also have advantages. L2 glosses are more easily provided when learners who have different L1 backgrounds. Furthermore, L2 glosses increase the amount of input in the target language. One shortcoming of L2 glosses, however, requires teachers to ensure that every word in the gloss is clearly comprehensible for all learners. This increases the burden on teachers. Failing to control the comprehensibility of glosses cancels out the benefit of glossing.
Earlier studies comparing the effects of L1 and L2 glosses (Jacobs et al., Reference Jacobs, Dufon and Hong1994; Ko, Reference Ko2012; Miyasako, Reference Miyasako2002; Yoshii, Reference Yoshii2006) have presented inconsistent findings. Jacobs et al. (Reference Jacobs, Dufon and Hong1994), Ko (Reference Ko2012), and Yoshii (Reference Yoshii2006) found no clear difference between L1 and L2 glosses. In contrast, Fang (Reference Fang2009), Xu (Reference Xu2010), and So (Reference So2010) found an advantage of L1 glosses over L2 glosses. Furthermore, different gloss languages might affect the retention of words differently. Yoshii (Reference Yoshii2006) found that scores on a meaning recall test showed different forgetting rates for words glossed in the L1 and L2 and indicated that the vocabulary gains in the L1 gloss condition declined more sharply over time than the L2 gloss condition.
Combining both the L1 and L2 in glosses is another approach. Some studies have investigated the effect of L1 plus L2 glosses in comparison to L1 and L2 glosses on their own (Azari, Abdullah, Heng, & Hoon, Reference Azari, Abdullah, Heng and Hoon2012; Ko, Reference Ko2017; Salehi & Naserieh, Reference Salehi and Naserieh2013; Xu, Reference Xu2010). The results of earlier studies are again inconsistent. Ko (Reference Ko2017) and Azari et al. (Reference Azari, Abdullah, Heng and Hoon2012) found that L1 plus L2 glosses were superior to either L1 or L2 glosses alone. Xu (Reference Xu2010) found that L1 glosses led to more learning than L1 plus L2 glosses on an immediate posttest, but on a delayed posttest, the L1 plus L2 glosses contributed to larger gains than L1 glosses. In contrast, Salehi and Naserieh (Reference Salehi and Naserieh2013) did not find any advantage of L1 plus L2 glosses over L1 glosses or over L2 glosses. However, all these conditions yielded greater learning than a nonglossed condition.
Many researchers argue that learners’ L2 proficiency may moderate the effectiveness of the language used for glossing (Ko, Reference Ko2012, Reference Ko2017; Yoshii, Reference Yoshii2006). For example, Nation (2013, p. 246) suggests that L1 glosses should be provided for beginners, while either L1 or L2 would work for advanced learners. Ko (Reference Ko2017) tested the effectiveness of three different conditions with a control condition (i.e., L1 gloss, L2 gloss, and L1 plus L2 gloss, no-gloss) with Korean EFL learners. The findings indicated that the L1 and L1 plus L2 glosses were effective for lower proficiency learners, while the L2 and L1 plus L2 glosses were effective for higher proficiency learners. However, because few individual studies directly examined the interaction between gloss languages and students’ proficiency levels, it is difficult to reach a conclusion just by looking at the results of individual studies. Alternatively, meta-analysis may allow us to investigate the effects of L2 proficiency level on the language of the gloss, by looking at the results of multiple studies that have recruited participants with different proficiency levels.
GLOSS MODALITIES
Glosses can be presented in different modalities. Widely used modalities include text, picture, video, auditory, and the combination of one or more of these modalities. The majority of the studies demonstrated that pictorial and textual glosses were equally effective (Kost et al., Reference Kost, Foss and Lenzini1999; Yanguas, Reference Yanguas2009; Yoshii & Flaitz, Reference Yoshii and Flaitz2002). Rassaei (Reference Rassaei2017) compared textual glosses and audio glosses. The results showed that audio glosses yielded greater learning than textual glosses both on the productive recall test and recognition test regardless of test timings.
Studies have also focused on whether combining different modalities (usually textual and pictorial) enhances the glossing effect (Boers, Warren, He, & Deconinck, Reference Boers, Warren, Grimshaw and Siyanova-Chanturia2017; Chun & Plass, Reference Chun and Plass1996; Warren, Boers, Grimshaw, & Siyanova-Chanturia, Reference Warren, Boers, Grimshaw and Siyanova-Chanturia2018; Yeh, Wang, & Tsing, Reference Yeh, Wang and Tsing2003). Yun’s (Reference Yun2011) and Vahedi et al.’s (Reference Vahedi, Ghonsooly and Pishghadam2016) earlier meta-analyses on the effect of glossing on vocabulary focused on the effects of multimodal glosses compared to single-mode glosses. However, the comparison between single-mode glosses and multimodal glosses does not indicate which specific gloss mode is more advantageous for vocabulary learning. Meta-analyzing earlier studies examining effects of single-mode glosses may shed light on the relative effectiveness of different types of gloss modes.
RATIONALE OF THE PRESENT STUDY
Due to the inconsistency of results in the literature, there is little consensus on how glosses should be provided. One approach to disentangling this incongruity is to conduct a meta-analytic review. Several attempts to synthesize studies investigating the effects of glossing on vocabulary learning from reading have already been made in earlier meta-analyses (Abraham, Reference Abraham2008; Vahedi et al., Reference Vahedi, Ghonsooly and Pishghadam2016; Yun, Reference Yun2011; see also Taylor, Reference Taylor2006, Reference Taylor2009, Reference Taylor2013, and 2014 for effects of glossing on reading comprehension). Abraham (Reference Abraham2008) synthesized 11 studies to investigate the overall effects of CALL (computer-assisted language learning) glosses on vocabulary learning and reading comprehension. Abraham examined the influence of moderator variables: proficiency, text type, and test. The study did not investigate how different gloss types, language, and modes affected vocabulary learning. Additionally, as Abraham notes, the meta-analysis results were based on a small number of studies (only six studies were analyzed as to the glossing effect on vocabulary learning), weakening the statistical power to test moderator variables and the reliability of its results.
Vahedi et al. (Reference Vahedi, Ghonsooly and Pishghadam2016) and Yun (Reference Yun2011) investigated the effect of multimodal (textual plus pictorial) glosses compared to single-mode (textual) glosses. Both of the studies compared treatment groups accessing multiple hypertext glosses with control groups accessing single-mode glosses in vocabulary learning through reading and found that multimodal glosses were found to be moderately more effective than single-mode glosses (Hedges’s g = 0.46 in Vahedi et al.; and Hedges’s g = 0.84 in Yun). These studies indicated that combining different gloss modes are more effective than single-mode glosses; however, we still do not know the extent to which single-mode glosses promote learning in its own right. This is surprising because it seems important to first understand the strengths and weaknesses of single-mode glosses before exploring the complex effects of the combinations of different gloss modes (i.e., multimodal glosses). Therefore, in our meta-analysis, we focused on various formats of single-mode glosses.
All three earlier meta-analyses on glossing (i.e., Abraham, Reference Abraham2008; Vahedi et al. Reference Vahedi, Ghonsooly and Pishghadam2016; Yun, Reference Yun2011) noted that their results may suffer from weak statistical power and reliability due to the small number of studies included. To increase the number of studies included, we used relative learning gain (proportion of unknown words learned) as standardized effect sizes (ESs) by following Swanborn and de Glopper’s (Reference Swanborn and de Glopper1999) meta-analysis of incidental L1 vocabulary learning from reading. This allows more studies to be included in the analysis because we can include not only studies that employed true control groups (i.e., participants who read texts without glosses) but also studies that compared different types of glosses (e.g., studies comparing L1 glosses and L2 glosses) without having true control groups. Using a relative learning gain also provides more comprehensible ESs (Swanborn & de Glopper, Reference Swanborn and de Glopper1999) representing the proportions of unknown words learned by participants with and without the provision of glossing. Furthermore, to account for variances between studies, this meta-analysis adopted three-level meta-regression (de Vos, Schriefers, Nivard, & Lemhöfer, Reference de Vos, Schriefers, Nivard and Lemhöfer2018; Lee, Warschauer, & Lee, Reference Lee, Warschauer and Lee2018; Van den Noortgate, López-López, Marín-Martínez, & Sánchez-Meca, Reference Van den Noortgate, López-López, Marín-Martínez and Sánchez-Meca2015). This allows the examination of differences between different forms of glossing used within each individual study as well as differences of different forms of glossing aggregated across all included studies. This analysis provides a more reliable estimation of the relative effectiveness between different glossing formats (e.g., Konstantopoulos, Reference Konstantopoulos2011).
Meta-analysis also allows the examination of variables that were not the focus of each individual study. By conducting moderator analyses, we examined the relationships between learning gains and the features of studies, such as the characteristics of reading materials and participants. Investigating these variables may offer a more precise and transparent picture of how glosses affect learning in different contexts. Additionally, by including test format as a variable, this study may provide a more accurate assessment of the extent to which vocabulary knowledge is gained through glossed reading.
The current study was guided by the following five research questions.
1. What is the overall effect of glossed reading on L2 incidental vocabulary learning from reading?
2. What are the relative effects of gloss type?
3. What are the relative effects of gloss language?
4. What are the relative effects of gloss mode?
5. To what extent do text and learner characteristics moderate glossing effects?
METHOD
LITERATURE SEARCH
We used several databases to search for studies to include in the meta-analysis—Education Resources Information Centre (ERIC), PsycINFO, Linguistics and Language Behavior Abstracts (LLBA), ProQuest Global Dissertations, and Google Scholar—by using various combinations of the following keywords: gloss, glosses, glossing, vocabulary, reading, annotation, annotate, annotating, L2, foreign, and second. This process identified 1,001 reports published until and including August 2018, all of which were retrieved and carefully reviewed.
To ensure the comprehensiveness of the literature search, we also inspected the reference sections of the previous meta-analyses (Abraham, Reference Abraham2008; Vahedi et al., Reference Vahedi, Ghonsooly and Pishghadam2016; Yun, Reference Yun2011), review articles on glossing effects on vocabulary learning (Azari, Reference Azari2012; Mohsen & Balakumar, Reference Mohsen and Balakumar2011), and studies identified through the electronic database search. Studies with relevant topics were also retrieved.
INCLUSION AND EXCLUSION CRITERIA
The following seven criteria were employed to determine which studies to include in the analysis.
1. Experimental or quasi-experimental studies investigating glossing effects on L2 vocabulary learning by comparing reading with glossing groups and a reading without glossing group, or by comparing different types of single-mode glossing were included.
2. Studies in which participants read a text or multiple texts were included. Studies focusing on sentence reading were not included.
3. Studies that used a between-participants design were included. Studies that used a within-participants design (e.g., Al-Seghayer, Reference Al-Seghayer2001) were excluded due to the limited number of studies using a within-participants design. This decision was based on the suggestion that blending two different research designs in a meta-analysis tends to produce biased results (Plonsky & Oswald, Reference Plonsky and Oswald2015).
4. Studies that only used unknown words as target words or controlled participants’ prior knowledge of the target words either by using nonsense words or pretest scores were included. We excluded studies reporting that participants already knew a considerable number of the target words before the treatment by administering pretests, but did not report the pretest scores (e.g., Jalali & Neiriz, Reference Jalali and Neiriz2012).
5. Studies that focused on incidental vocabulary learning (participants were not forewarned about vocabulary posttests, Hulstijn, Reference Hulstijn and Robinson2001) were included. Studies in which participants were told that they would be tested on vocabulary or explicitly told to learn vocabulary (e.g., Holley & King, Reference Holley and King1971), as well as studies where participants were repeatedly tested before and after the reading sessions (e.g., AbuSeileek, Reference AbuSeileek2013) were excluded. As such, we aimed to exclude studies in which there was the possibility that participants focused on remembering glossed target words instead of reading texts for comprehension (Hulstijn, Reference Hulstijn, Arnaud and Béjoint1992). One study in which participants engaged in word-focused activities (i.e., So, Reference So2010) was also excluded.
6. Studies reporting enough statistical information (i.e., mean, SD, number of participants tested) required for meta-analysis were included.
7. Articles written in English were included.
8. Studies were excluded when different target words were used in the different learning conditions, or there was insufficient information to interpret the methodologies (e.g., not reporting how vocabulary knowledge was measured).
9. Studies that focused on the effect of corpus-based glosses (e.g., Lee, Warschauer, & Lee, Reference Lee, Warschauer and Lee2017) were excluded. This is because corpus-based glosses provide corpus-extracted sentences for glossed words and require readers to infer the meanings of target words, which is quite different from other gloss types where form-meaning connections are provided.
Given the inclusion of unpublished studies (e.g., MA and PhD theses, conference presentations), securing the quality of the included studies is vital. We strictly excluded studies that failed to clearly describe how learning gains were measured or how participants’ responses on tests were scored. Forty-two studies (N = 3802) providing 359 posttest scores satisfied all the criteria (see Online Supplementary Material Appendix 2 for the flowchart for inclusion and exclusion of the studies following the PRISMA 2009 flow diagram; Moher, Liberati, Tetzlaff, Altman, & Group, Reference Moher, Liberati, Tetzlaff, Altman and Group2009). These 42 studies comprised 31 journal articles, 3 conference presentations, 3 bulletin reports, 3 PhD theses, 1 MA thesis, and 1 online report (see Online Supplementary Material Appendix 3 for basic information about the included studies).
CODING
All 42 studies meeting the selection criteria were coded for several variables: outcome variables, glossing variables, textual and learner characteristics, as well as study identifier (e.g., author, year, experiment number, participant group number).Footnote 1
Outcome Variables
For dependent variables, mean posttest scores, SDs, number of participants tested, and test timing (immediate or delayed) were coded. To account for the strength of vocabulary knowledge measured (Yanagisawa & Webb, Reference Yanagisawa, Webb and Webb2019), we followed earlier meta-analyses on L2 vocabulary learning (de Vos et al., Reference de Vos, Schriefers, Nivard and Lemhöfer2018; Uchihara, Webb, & Yanagisawa, Reference Uchihara, Webb and Yanagisawa2019) and coded test format as either: (a) recall (b) recognition, or (c) other tests. De Vos et al.’s (Reference de Vos, Schriefers, Nivard and Lemhöfer2018) meta-analysis of L2 vocabulary learning from spoken input grouped vocabulary tests into two types, recognition and recall, based on its sensitivity. Recognition tests included form recognition (knowledge of word forms, i.e., spellings), and receptive and productive recognition (knowledge of form-meaning links). Recall tests included receptive and productive recall (knowledge of form-meaning links). We added one more category, other tests, where participants’ vocabulary knowledge was tested beyond the form-meaning connection and involved measurement of participants’ use of target words. Fill-in-the-blank tests and the Vocabulary Knowledge Scale (VKS) were coded as other test formats.
Glossing Variables
Glossing was coded for type, language, and modality. Gloss type was coded as either marginal, multiple-choice, glossary, in-text, interlinear, hyperlinked, or no-gloss. There were no other types of glosses identified among the included studies. Gloss languages were coded as L1, L2, or L1 plus L2. Gloss modalities were coded as textual, pictorial, or auditory. Other gloss languages or modalities were not identified in the included studies.
Text Characteristics
Five variables were coded for text characteristics: L2 learner targeted material, comprehensibility, target word percentage, text type, and CALL use. For L2 learner targeted material, we coded texts as either targeted for L2 learners or native speakers. Reading materials were coded as L2 learner targeted material when they were written or edited by researchers and when researchers used texts of textbooks or workbooks written for L2 learners. When the title of the book (e.g., textbooks and workbooks) was reported, we searched on the Internet to determine whether the book was written for L2 learners or native speakers. When it was not clear whether the material was written for L2 learners or native speakers, the study was excluded from this analysis.
Some studies investigated participants’ reading comprehension using recall tests or multiple-choice questions. We coded and standardized the test scores (dividing the scores by the maximum score of each test then multiplying the resultant scores by 100), which indicates how well participants understood the reading material or the difficulty of the reading material.
Target word percentage was calculated by dividing the total number of target words by the total number of words participants read then multiplying by 100. Target word percentage can roughly be interpreted as the percentage of words that the researchers believed the learners did not know in texts.
Text type was coded as either narrative or expository. CALL use was coded as either yes (i.e., reading material was presented on a computer screen) or no (i.e., on paper).
Learner Characteristics
Learners’ L2 proficiency levels and institutional levels were coded. For learners’ proficiency, we coded them as beginner, intermediate, or advanced. Institutional level was coded as secondary school, university, or language school. No other institution was identified in the included studies.
Coding Procedure
Following earlier meta-analyses and suggestions (Plonsky & Oswald, Reference Plonsky and Oswald2015), two of the authors of this study engaged in the coding process to enhance the reliability of the coding. First, the two authors coded five studies separately using the coding scheme. The coding agreement between the two coders was calculated by using Cohen’s kappa and showed a high agreement rate at κ = .97. All discrepancies were discussed until achieving consensus. Finally, one of the authors carefully coded the rest of the studies, while the other author randomly coded 28.6% (12 out of 42) of the remaining studies. This time, there were no discrepancies between coders.
EFFECT SIZE CALCULATION
Following Swanborn and de Glopper’s (Reference Swanborn and de Glopper1999) meta-analysis on vocabulary learning, we adopted the proportion of unknown words learned as ESs using the following formula (see also, Horst et al., Reference Horst, Cobb and Meara1998; Webb & Chang, Reference Webb and Chang2015):
$${\rm{ES}} = {{{\rm{Mean\ posttest\ score}} - {\rm{Mean\ pretest\ score}}} \over {{\rm{Maximum\ posttest\ score}} - {\rm{Mean\ pretest\ score}}}}$$When studies used target words that were all unknown to participants, mean pretest score was set as 0. When studies used control group (i.e., a group that only took the same vocabulary test without any exposure to target words) to account for participants prior knowledge of target words, the control group’s mean posttest score was used as the mean pretest score (see Online Supplementary Material Appendix 4 for the detailed calculation formulas).
To appropriately weight ESs, we calculated the sampling variance using the reported SDs of posttest scores that were converted into proportions. The formula in Hox (2010, p. 209) was used, where s refers to a proportion-translated SD of posttest scores:
Sampling variance = 
${{{\rm{s}}^2 } \mathord{\left/ {\vphantom {{{\rm{s}}^2 } {\rm{n}}}} \right. \kern-\nulldelimiterspace} {\rm{n}}}$
In Swanborn and de Glopper (Reference Swanborn and de Glopper1999), only number of participants was used to calculate sampling variance while ignoring the test score variance (which is reported as SD or SE in individual studies). To obtain more precise information about the sampling variance to enhance the accuracy of the analysis, the current meta-analysis used reported SD of posttest scores to calculate sampling variance. The detailed analytic approach regarding ES and sampling variance calculation, as well as calculation formulas for ESs and sampling variances, can be found in the Online Supplementary Material Appendix 4.
Overall, the included 42 studies reported 359 posttest scores, all of which were transformed into ESs and included in the analysis.
ANALYSIS
As described previously, we converted all reported posttest scores into a relative learning gain (i.e., proportion of the unknown words learned) and used this as ESs. A relative gain is a widely used index to standardize the learning gain among the vocabulary research (e.g., Horst et al., Reference Horst, Cobb and Meara1998; Swanborn & de Glopper, Reference Swanborn and de Glopper1999; Webb & Chang, Reference Webb and Chang2015). Using relative learning gains allows us to compare learning gains from different conditions of different studies on the same scale. Furthermore, we used a three-level meta-analysis (e.g., Cheung, Reference Cheung2014; de Vos et al., Reference de Vos, Schriefers, Nivard and Lemhöfer2018; Lee et al., Reference Lee, Warschauer and Lee2018) to model three different sources of variances with sampling variance being the first level, variance between the different ESs from the same study being the second, and variance between studies being the third. The three-level meta-analysis is a technique that considers dependencies of ESs (i.e., which ESs are produced by the same studies) so that variances within studies as well as variances between studies are examined at the same time. This technique produces more reliable and robust estimations when examining the different glossing conditions compared within each study (e.g., Konstantopoulos, Reference Konstantopoulos2011). Furthermore, quite a few studies reported multiple ESs that are dependent due to sampling error variance (i.e., the same participants were tested with different measurements or repeatedly tested at different times). To appropriately deal with the potential Type I error inflation due to these dependent ESs, we applied the cluster-robust variance estimation (Hedges, Tipton, & Johnson, Reference Hedges, Tipton and Johnson2010) with small-sample adjustments (Tipton, Reference Tipton2015; Tipton & Pustejovsky, Reference Tipton and Pustejovsky2015). All the analyses were conducted in the R statistical environment (R Core Team, 2017) using the metafor package (Viechtbauer, Reference Viechtbauer2010) and the clubSandwich package (Pustejovsky, Reference Pustejovsky2018). An F test (i.e., Wald-test) was conducted when there was more than one level in a predictor variable included in a statistical model. ESs of immediate posttests and delayed posttests were analyzed separately. We set the significance level at 5%. P-values lower than .10 were also reported as of marginal significance. This did not mean to imply that the null hypothesis was rejected, but to report the trend of the data.
Analysis Procedure
The ESs of immediate posttests and delayed posttests were analyzed separately. For Research Question 1, to investigate the overall effect of glossed reading, meta-regression models with a gloss variable (gloss vs. nongloss) that predicts the ESs with and without intercept were fitted with the data. Then, to investigate whether the glossing effect varied based on test formats, the interaction term between the gloss variable and test format was added to the model.
For Research Questions 2, 3, and 4 comparing different formats of glosses (type, language, mode) to investigate whether different formats led to significantly greater learning than nongloss reading, meta-regression models with a gloss format variable (type, language, mode) that predicts ESs were administered. Test format was also inserted as a covariate to control for the potential bias due to the test format (e.g., less effective gloss types might not have been measured with more demanding tests). Nonglossed condition and recall test format were set as reference levels. Next, to investigate the relative effectiveness between different formats, multiple comparisons were administered by changing the reference levels (de Vos et al., Reference de Vos, Schriefers, Nivard and Lemhöfer2018).
For Research Question 5 investigating the effects of text and learner characteristics, to determine whether each variable correlated with ES, each predictor variable was inserted into a meta-regression model with a gloss variable (gloss vs. nongloss) and test format. Next, to investigate whether the effect of each variable changed based on having glosses or not, an interaction term between each variable and gloss variable was added to the first meta-regression model. To avoid biases caused by outliers, for analyzing numerical variables (i.e., comprehensibility and target word percentage), ESs for each variable deviated 2 SD from the mean were excluded.
Sensitivity Analyses
Potential publication biases and outliers of ESs influencing the overall ESs were checked using Egger’s tests (Egger, Smith, Schneider, & Minder, Reference Egger, Smith, Schneider and Minder1997), funnel plots, and Cook’s distance (de Vos et al., Reference de Vos, Schriefers, Nivard and Lemhöfer2018; Viechtbauer & Cheung, Reference Viechtbauer and Cheung2010). Examining funnel plots showed that an ES based on Zhao and Ren’s (Reference Zhao and Ren2017) immediate posttest score of the nonglossed reading condition, the low-proficiency group indicated a very small variance (SD = 0.08, maximum test score = 24) that greatly deviated from other ESs, so this participant group was excluded from the analysis. No other obvious publication biases or outliers were identified. We also analyzed whether learning gains or glossing effect influenced publication status. Those preliminary analyses did not find any publication biases. Furthermore, other methodological variables (e.g., participant allocation method, delayed posttest timing) were also examined. The detailed results of these diagnostic tests, as well as methodological variables, can be found in the Online Supplementary Material Appendix 5.
RESULTS
RESEARCH QUESTION 1: WHAT IS THE OVERALL EFFECT OF GLOSSED READING ON L2 INCIDENTAL VOCABULARY LEARNING FROM READING?
A total of 30 studies included nongloss reading conditions with 89 posttest scores, and 42 studies included glossed reading conditions with 259 posttest scores. Table 1 shows the overall effect of learning gains separately for glossed reading and nonglossed reading. For immediate posttests, glossed reading led to 45.3% (95% CI [38.7, 51.8]) of the unknown words were learned (b = 0.453, SE = 0.032, p < .001). Nonglossed reading yielded the learning of 26.6% (95% CI [20.5, 32.7]) of the unknown words (b = 0.266, SE = 0.030, p < .001), and this was significantly lower than glossed reading (b = ‒0.187, SE = 0.022, p < .001).
The overall effect of glossed reading
Note: k = number of studies, n = number of ESs, CI = 95% confidence interval, Mean ES = weighted mean effect sizes converted into a percentage from proportion for the sake of interpretability. p = p-value for significant test. Pseudo-R Footnote 2 (Raudenbush, Reference Raudenbush, Cooper, Hedges and Valentine2009) showed that 11% and 9.4 % of the variance in the ESs was explained for the immediate and the delayed posttests, respectively.
For delayed posttest scores, glossed reading led to the learning of 33.4% (95% CI [27.0, 39.8]) of the unknown words (b = 0.334, SE = 0.032, p < .001). Nonglossed reading led to the learning of 19.8% (95% CI [13.1, 26.5]) of unknown words (b = 0.198, SE = 0.033, p < .001). The difference between glossed reading and nonglossed reading was significant (b = ‒0.136, SE = 0.021, p < .001), showing that the glossing effect was retained through to the delayed posttests.
How Do Learning Gains Differ in Relation to Test Format?
Learning gain from reading may vary significantly across the different types of vocabulary measurements used in each study. First, to determine whether vocabulary learning from reading, in general, differs in the format of measurements, we entered the main effects of gloss and test format into a meta-regression model (see Table 2 for the mean ESs for each test format). Second, to determine whether glossing promoted the learning of a specific aspect of vocabulary knowledge, the interaction term between gloss and test format was added.
The overall effect of glossed reading: Separately for each test format
Note: k = number of studies, n = number of ESs, CI = 95% confidence interval, Mean ES = weighted mean effect sizes converted into a percentage from a proportion for the sake of interpretability. p = p-value for significant test.
The analysis of immediate posttests revealed that the main effects of gloss and test format were significant, F(21.7) = 67.5, p < .001, F(9.46) = 6.53, p = .017, respectively, but the interaction was nonsignificant, F(8.34) = 1.17, p = .356. Subsequent multiple comparisons using the model without the interaction term showed that recognition tests led to a 24.8% higher ES than recall tests (b = 0.248, p = .003). Other tests (i.e., VKS and gap-filling tests) led to a 20.5% higher mean ES than recall tests, which was significant (b = 0.205, p = .045). There was no significant difference between other tests and recognition tests (p = .589).
The analysis of delayed posttests revealed that the interaction between gloss and test format was significant, F(12) = 4.41, p = .037, indicating that glossing effects differed depending on which tests were administered. Subsequent multiple comparisons using the model with interaction term showed that the glossing effect was 9.3% higher when measured with recognition tests compared to when measured with recall tests (b = 0.093, p = .008). There was no significant difference in gloss effects between other tests and recall tests (p = .574), or other tests and recognition tests (p = .127).
RESEARCH QUESTION 2: WHAT ARE THE RELATIVE EFFECTS OF GLOSS TYPES?
The number of studies including nonglossed conditions was 30, marginal glosses = 26, multiple-choice glosses = 13, hyperlinked glosses = 12, in-text glosses = 4, glossaries = 3, and interlinear glosses = 1. Test format was also inserted as a covariate to control for the potential bias due to the test format (e.g., less effective gloss types might not have been measured with more demanding tests).
The analysis of immediate posttests showed that different gloss types uniquely contributed to vocabulary learning. Table 3 shows the difference between each gloss type and the nonglossed condition. Multiple-choice led to the greatest gain, followed by marginal, hyperlinked, glossaries, interlinear, and in-text glosses in that order. While multiple-choice, hyperlinked, marginal, and interlinear glosses led to significantly greater learning compared to nongloss (p < .05) and in-text glosses approached statistical significance (p = .055), glossaries did not reach the statistical significance (p = .134). Subsequent multiple comparisons revealed that multiple-choice glosses contributed to significantly higher scores than in-text glosses (b = 0.142, p = .026), interlinear (b = 0.092, p = .026), and marginal (b = 0.074, p = .035). Interlinear glosses were significantly higher than in-text glosses (b = 0.050, p = .044) and significantly lower than marginal glosses (b = 0.142, p = .023), but note that only four ESs from Zarei and Hasani (Reference Zarei and Hasani2011) accounted for interlinear glosses.
The learning gain for each gloss type compared to the nonglossed condition
Note: k = number of studies, n = number of ESs, CI = 95% confidence interval, Mean ES difference (%) = mean effect size differences between each gloss type and the nonglossed condition converted into a percentage. p = p-value for significant test.
The analysis of delayed posttests revealed a slightly different pattern for each gloss type. The most effective gloss type on delayed posttests was multiple-choice, followed by hyperlinked, marginal glosses, glossaries, and in-text glosses in that order. In-text glosses and glossaries did not significantly differ from nongloss (p = .147, p = .412, respectively). Multiple comparisons did not find any significant differences between each of the other gloss types.
RESEARCH QUESTION 3: WHAT ARE THE RELATIVE EFFECTS OF GLOSS LANGUAGES?
Among the included studies, 31 included the L1 gloss condition, 25 included L2 glosses, 3 included L1 plus L2 glosses, and 30 studies included nonglossed conditions. Table 4 shows the difference between each gloss language and nonglossed conditions. All gloss languages led to significantly greater learning gains compared to the nonglossed condition. L1 plus L2 led to the highest gain followed by L1, and L2 glosses in that order. Multiple comparisons revealed that L1 glosses contributed to 4% higher gains than L2 glosses (b = 0.040, p = .075). There were no significant differences between L1 plus L2 glosses and L1 (p = .367) and between L1 plus L2 glosses and L2 (p = .139).
The learning gain for each gloss language compared to the nonglossed condition
Note: k = number of studies, n = number of ESs, CI = 95% confidence interval, Mean ES difference (%) = mean effect size differences between each gloss language and the nonglossed condition converted into a percentage. p = p-value for significant test.
The analysis of delayed posttests showed a similar trend as the results of immediate posttests. Every gloss language led to significantly greater learning gains than the nonglossed condition. Although the order of effectiveness was slightly different from the results of immediate posttests (L1, L1 plus L2, and L2 glosses), multiple comparisons found the same results as immediate posttests; that there was a statistical significance between L1 and L2 glosses (b = 0.052, p = .048). There were no significant differences between L1 plus L2 glosses and L1 (p = .962) and between L1 plus L2 glosses and L2 (p = .138).
RESEARCH QUESTION 4: WHAT ARE THE RELATIVE EFFECTS OF GLOSS MODES?
Forty-two studies included the textual gloss condition, 5 studies included pictorial glosses, 2 studies included audio glosses, and 30 studies included nonglossed conditions. Table 5 shows the difference between each gloss mode and no-gloss. For immediate posttests, every gloss mode led to significantly greater learning gains compared to the nonglossed condition. Auditory glosses led to the highest ES, followed by pictorial glosses, then textual glosses. However, this result should be interpreted with caution because the ESs of auditory glosses came from only two studies (i.e., Rassaei, Reference Rassaei2017; Sadeghi & Ahmadi, Reference Sadeghi and Ahmadi2012). Multiple comparisons did not find any significant differences across different gloss modes.
The learning gain for each gloss mode compared to the nonglossed condition
Note: k = number of studies, n = number of ESs, CI = 95% confidence interval, Mean ES difference (%) = mean effect size differences between each gloss mode and the nonglossed condition converted into a percentage. p = p-value for significant test.
The analysis of delayed posttests showed a similar trend as the results of immediate posttests. Auditory glosses led to the greatest ES; however, it did not reach statistical significance when compared with the nonglossed condition (p = .108). This may be due to the small sample size (only two studies accounted for auditory glosses). Pictorial and textual glosses led to greater learning gains than no-gloss (p = .023, p < .001, respectively). Multiple comparisons found that auditory glosses were marginal significantly higher than pictorial glosses (b = 0.221, p = .091). No significant differences were found between textual and auditory glosses (p = .177), or between textual and pictorial glosses (p = .396).
RESEARCH QUESTION 5: TO WHAT EXTENT DO TEXT AND LEARNER CHARACTERISTICS MODERATE GLOSSING EFFECTS?
Text Characteristics
Regarding the L2 learner targeted material variable, in 29 studies, participants read texts that targeted L2 learners. In five studies, participants read texts that were written for native speakers. Neither of the analyses on immediate nor delayed posttests found any significant main effect (p = .229, p = .137, respectively) or interaction with glossing (p = .223, p = .540, respectively). This indicates that the audience that material was targeted for was not clearly associated with learning gains or glossing effects.
As for comprehensibility of texts, 16 studies reported participants’ reading comprehension scores. After excluding six ESs from Jacobs et at. (1994) identified as outliers, 140 ESs were included into the analysis.Footnote 2 The analysis of immediate posttests showed that comprehensibility was significantly associated with ESs (b = 0.004, p = .018) while the effect of glossing was controlled. This indicates that the estimated learning gain increases by 4% as the comprehension test score increases by 10% even when the effect of glossing is controlled. The interaction between comprehensibility and glossing was not significant (p = .437), which implies that reading comprehension and glossing independently influenced vocabulary learning from reading (i.e., glossing may promote vocabulary learning regardless of how well learners comprehend texts). The same trend was found on delayed posttests; comprehensibility was significantly associated with ESs (b = 0.004, p = 0.09) while the effect of glossing was controlled, and their interaction was not significant (p = .252).
For target word percentage, the mean target word percentage was 2.7% (SD = 1.8%, Min = 0.6%, Max = 7.7%). Neither of the analyses on immediate nor delayed posttests found any significant main effect (p = .938, p = .898, respectively) or interaction with glossing (p = .716, p = .866, respectively).
For text type, 18 studies used expository texts, and 17 studies used narrative texts. Neither of the analyses on immediate nor delayed posttests found any significant main effect (p = .472, p = .800, respectively) or interaction with glossing (p = .139, p = .171).
For CALL use, in 28 studies, participants read texts on a computer screen, while in 16 studies, participants read texts on paper. Neither analysis on immediate nor delayed posttests found any significant main effect (p = .614, p = .491, respectively) or interaction with glossing (p = .942, p = .850, respectively).
Learner Characteristics
Out of 42 studies, 25 studies reported learners’ L2 proficiency levels: 9 studies recruited beginner learners, 16 studies involved intermediate learners, and 3 studies recruited advanced learners. Studies reported participants’ L2 proficiency based on different references. Eight studies (32%) referred to participants’ English proficiency tests (e.g., International English Language Testing System [or IELTS], Oxford Placement Test, Test of English for International Communication [or TOEIC]), two studies (8%) used cloze tests, two studies (8%) looked at the context that participants were in (i.e., textbook used in the classroom, school administration’s report), two studies (8%) looked at English proficiency tests and vocabulary size tests, and one study (4%) only used vocabulary size tests. The rest of the studies (10 studies, 40%) did not report how they determined proficiency.
The analyses of immediate and delayed posttests found that there was no significant main effect (p = .262, p = .331, respectively), suggesting that overall vocabulary learning gains did not significantly differ based on the how proficient students were. An interaction was significant on immediate posttests (p = .048), showing that the glossing effect on vocabulary learning was moderated by learners’ proficiency. Subsequent multiple comparisons of immediate posttests revealed that intermediate learners benefitted from glosses significantly more than beginners (b = 0.125, p = .026) and advanced learners (b = 0.187, p = .055). There was no significant difference between beginner and advanced learners (p = .395). The interaction was not significant on delayed posttests (p = .145); however, the same trend was observed, intermediate learners benefitted from glosses the most, followed by the beginners and advanced learners.
Do L2 proficiency levels benefit differently from different gloss languages? Because proficiency was not always reported, L1 plus L2 glosses suffered from a small number of studies (i.e., less than three) for each proficiency level. Hence, we focused on the L1 glosses and L2 glosses. For L1 glosses, 8 studies included beginners, 11 studies included intermediate level learners, and 3 studies included advanced learners. For L2 glosses, 8 studies included beginners, 11 studies included intermediate learners, and 3 included advanced learners. For nongloss, 6 studies included beginners, 11 included intermediate learners, and 3 included advanced learners.
The analyses of immediate and delayed posttests did not find any significant interaction between gloss languages and proficiency (p = .430, p = .340, respectively), suggesting that L1 glosses yielded greater learning than L2 glosses, regardless of participants’ L2 proficiency. However, this result should be interpreted with caution because a small number of studies included advanced learners and only a few studies compared different languages with students at different proficiency levels.
For participants’ institutional levels, 31 studies recruited university students, 6 studies included language school students, and 5 studies recruited secondary school students. The analyses of immediate and delayed posttests found that there were no significant main effects (p = .398, p = .914, respectively) or interactions (p = .242, p = .150, respectively). However, there was a trend on the delayed posttests that language school students benefitted more from glossing more than secondary school (b = 0.183, 95% CI [0.338, 0.273], p = .030) and university students (b = 0.170, 95% CI [0.396, ‒0.055], p = .088).
Moderator analysis for text and learner characteristics on immediate posttests
Note: k = number of studies. n = number of ESs. –ref.– = reference level. Coef. = estimated coefficient. p = p-value for significant test. CI = 95% confidence interval. Main effect refers to whether the moderator variable was related with the relative learning gains regardless of gloss provision, and interaction effect refers to whether the moderator variable mediated the glossing effect (i.e., to what extent the mean learning gain through glossed reading differed from nonglossed reading).
Moderator analysis for text and learner characteristics on delayed posttests
Note: k = number of studies. n = number of ESs. –ref.– = reference level. Coef. = estimated coefficient. p = p-value for significant test. CI = 95% confidence interval. Main effect refers to whether the moderator variable was related with the relative learning gains regardless of gloss provision, and interaction effect refers to whether the moderator variable mediated the glossing effect (i.e., to what extent the mean learning gain through glossed reading differed from nonglossed reading).
DISCUSSION AND CONCLUSION
The current meta-analysis revealed that, on average, learners reading texts with glosses learned 45.3% of the unknown words on immediate posttests, and 33.4% on delayed posttests. These rates were significantly higher than learners who read texts without glosses (26.6% for immediate and 19.8% for delayed posttests). Glossing was found to contribute to vocabulary learning across all types of tests (i.e., recognition, recall, other). Glossing effects on delayed posttest scores were 9.3% smaller when measured with recall tests compared to when measured with recognition tests. This result suggests that learners may quickly become unable to recall target words they learned with glosses but most of them remain recognizable to a greater extent. These findings support earlier studies demonstrating that word recall is more difficult to acquire than recognition (González-Fernández & Schmitt, Reference González-Fernández and Schmitt2019; Laufer & Goldstein, Reference Laufer and Goldstein2004). In addition to word recognition and recall, knowledge required for other tests (VKS and gap-filling tests) was improved by glossing, indicating that glossed reading not only increases L2 learners’ form-meaning connection but might also enhance knowledge required to use words.
DIFFERENT APPROACHES TO GLOSSING
Gloss Type
The analysis of gloss type revealed that multiple-choice glosses were the most effective gloss type (mean ESs were 25.2% and 15.6% higher than nonglossed reading on immediate posttests and delayed posttests, respectively). Hyperlinked (18.4%, 15.2%), marginal (17.8%, 12.8%), and interlinear glosses (16.0% on immediate posttest) led to somewhat similar effects. The effectiveness of glossaries (17.4%, 10.4%) and in-text glosses (11.0%, 6.5%) were not so clear; when compared to nonglossed reading, no significant difference was found on either immediate or delayed posttests.
The advantage of multiple-choice glosses over the other gloss types was distinct especially on immediate posttests. The characteristics of multiple-choice glosses, where learners have to read sentences to select the appropriate choice, which may have led to deeper processing and greater learning gains. The superiority of multiple-choice glosses may be explained by deeper processing (Hulstijn, Reference Hulstijn, Arnaud and Béjoint1992; Rott, Reference Rott2005); the multiple-choice glosses require learners to read the sentence carefully and select the appropriate gloss among options. This extra processing may have strengthened the form-meaning connections of words. This finding is also in line with theories of vocabulary learning, mental effort hypothesis (Hulstijn, Reference Hulstijn, Arnaud and Béjoint1992) and involvement load hypothesis (Laufer & Hulstijn, 2001). Additionally, learners’ skipping behavior may also explain the effect of multiple-choice glosses. Many studies found that students often ignore unknown words even when glosses were provided (e.g., Boers, Warren, Grimshaw, & Siyanova-Chanturia, Reference Boers, Warren, Grimshaw and Siyanova-Chanturia2017; Hulstijn et al., Reference Hulstijn, Hollander and Greidanus1996; Warren et al., Reference Warren, Boers, Grimshaw and Siyanova-Chanturia2018). In contrast, in multiple-choice gloss conditions, students are told to look at each gloss to make a choice, so glossed words are not skipped. This enforced processing may lead to a greater learning rate.
Several factors potentially influence the effectiveness of multiple-choice glosses. For example, numbers of options may impact the degree to which learners evaluate each option for the sentence before selecting the best option (Hulstijn, Reference Hulstijn, Arnaud and Béjoint1992). Too many options might lead to more wrong selections and less learning, while too few options might hinder learning as learners easily choose the best option without deepening the processing of words. Additionally, the difficulty involving in selecting appropriate glosses may also be influenced by the similarities of options and participants’ L2 proficiency levels. Lastly, some studies examined whether providing feedback for learners’ selection enhances learning; however, the results are so far inconsistent (Nagata, Reference Nagata1999; Yoshii, Reference Yoshii2013). Multiple-choice glosses potentially hinder comprehension of the text (Martínez-Fernández, Reference Martínez-Fernández, Bowles, Foote, Perpiñán and Bhatt2008; however, see also Rott, Reference Rott2005). Studies directly looking at factors influencing the effectiveness of multiple-choice glosses are scarce, warranting further research exploring how multiple-choice glosses should be implemented.
The second most effective group of gloss types—that is, marginal and hyperlinked—may benefit from their location. They are neither too close to nor too far from the target word. Marginal and hyperlinked glosses are close enough for learners to quickly check meanings of target words, which may provide more opportunities to make form-meaning links between unknown words and their meanings. In contrast, glossaries tend to be located far from target words; learners usually have to look at the end of the reading material to check a glossary. This might discourage learners from checking, potentially decreasing the frequency of consulting a glossary. In-text glosses could be too close to target words, as in-text glosses are provided right after each target word. Because learners can easily comprehend the message of the sentence just by looking at an in-text gloss, the target word does not necessarily have to be processed (Watanabe, Reference Watanabe1997). It is reasonable to speculate that learners might have ignored glosses or target words given that all included studies in this meta-analysis set up incidental vocabulary learning conditions (i.e., participants were not told to learn target words) where comprehension of the text was the main purpose of reading. Participants may, therefore, have had no reason to check all glosses unless they had difficulty comprehending a text.
Only one study included interlinear glosses, which makes it difficult to draw a firm conclusion. Its effectiveness reached the statistical significance; however, the mean ES was at the fifth place out of six. Similarly, the number of studies including glossaries and in-text glosses was relatively small (2‒4). This could be because researchers would be more likely to hesitate to investigate the less effective gloss types based on their personal experience or intuitive reasoning. To arrive at a more robust conclusion, it would be useful to conduct more studies to examine the relative effectiveness of gloss types that are reported as less effective in this meta-analysis as well as ones reported as more effective.
Gloss Language
The analysis of gloss language showed that L1 glosses and L1 plus L2 glosses led to similar learning gains. Reading with L2 glosses led to the smallest learning gain. This trend was consistent across immediate and delayed posttests. These findings indicate that unknown target words are more easily learned in glosses with L1 translations compared to L2 definitions or synonyms in general. Newly learned L2 words are typically mapped onto their L1 translation (Clenton, Reference Clenton2015; Jiang, Reference Jiang2002; Kroll & Stewart, Reference Kroll and Stewart1994; Kroll, Van Hell, Tokowicz, & Green, Reference Kroll, Hell, Tokowicz and Green2010). Because L1 words tend to be more familiar to learners than L2 words, connections between L1 words and unknown L2 words can be more easily established compared to connections between L2 words and unknown L2 words (Choi, Reference Choi2016). Students’ skipping behavior could also explain the superiority of providing L1 glosses. Bell and LeBlanc (Reference Bell and LeBlanc2000) found that participants more frequently consulted L1 glosses than L2 glosses. Boers et al. (Reference Boers, Warren, Grimshaw and Siyanova-Chanturia2017) and Warren et al. (Reference Warren, Boers, Grimshaw and Siyanova-Chanturia2018) tracked learners’ eye movements while reading glossed tests and found that unknown words annotated with L2 textual glosses were about twice as frequently ignored than pictorial glosses or glosses combining L2 textual and pictorial glosses. These findings point to the possibility that looking up unknown words in L2 glosses can be demanding and demotivate students from using glosses while reading. L1 glosses, in contrast, may be looked up more frequently than L2 glosses and lead to greater learning gains.
Regarding L2 proficiency, we did not find any support for the hypothesis that students with higher proficiency levels benefit more from L2 glosses than those with lower proficiency levels. There are two possible explanations for this. The first possibility is that the proficiency of participants may not have been high enough to benefit from L2 glosses as much as L1 glosses. Another possibility relates to types of studies accounting for advanced learners in this meta-analysis. Among the total of three studies, except for Choi (Reference Choi2016), none of the studies compared L1 and L2 glosses. To confirm the potential interaction between proficiency and gloss languages, more research recruiting advanced L2 learners is required.
It is probably worth emphasizing that although the results suggested an advantage of L1 glosses over L2 glosses, the effect of L2 glosses was clearly observed when compared to reading without glosses. Given the fact that L2 glosses are very useful especially for contexts where each learner’s L1 background differs, the effectiveness of L2 glosses should not be disregarded.
Gloss Mode
The analysis of gloss mode did not find clear differences between the different modes of glossing. Interestingly, the mean ES for auditory glosses was larger than for textual glosses and pictorial glosses. However, the difference was not statistically significant. These comparisons have to be interpreted with caution because only two studies included auditory glosses (Rassaei, Reference Rassaei2017; Sadeghi & Ahmadi, Reference Sadeghi and Ahmadi2012). Both studies reported that auditory glosses led to greater learning than textual glosses. Rassaei (Reference Rassaei2017) argued that the advantage of auditory glosses could be due to the fact that glosses were provided in a different channel than the one was used for reading (i.e., textual), which potentially allowed learners to direct more attention to the glosses or pay attention to glosses while looking at glossed words. Further research to evaluate the effect of auditory glosses is warranted.
TEXT CHARACTERISTICS
Moderator analyses of text characteristics found that comprehension was a significant variable; reading comprehension and glossing independently promoted vocabulary learning. This result indicates that when learners read texts that are easier to understand, they learn more vocabulary from them. The results also imply that glossing promotes vocabulary learning even with reading materials that are relatively difficult for learners to comprehend. However, given that comprehensibility enhances vocabulary learning, it may be important to provide texts at the appropriate level to learners to maximize vocabulary learning even when providing glosses.
Other text-related variables, L2 learner target materials, text type, and CALL use, were not significantly related to ESs or glossing effects. Lack of clear effects of L2 learner target materials may indicate that learning gains or glossing effects do not differ so much by whether or not reading materials are written for native speakers or L2 learners as long as the difficulty level of them are appropriate for learners. Studies using materials written for native speakers may have included relatively advanced learners.
Lack of a clear advantage of CALL use seems counterintuitive at first glance. Taylor’s series of meta-analyses (2006, 2009, 2013, 2014) focused on the effects of glosses on L2 reading comprehension and revealed the advantage of CALL glosses over paper-based glosses. One reason is that CALL glosses might contribute to vocabulary learning in a different mechanism from reading comprehension. Another potential explanation is that this meta-analysis operationalized CALL use as whether reading material was presented on a computer screen (as opposed to paper) without considering how the glosses were provided (e.g., gloss type and language). Providing glosses in a CALL context does not necessarily lead to better vocabulary learning compared to reading on paper, and the type and language of glosses may impact more on the effectiveness of glossing.
LEARNER CHARACTERISTICS
The results showed that glossing effects differed in relation to L2 proficiency levels. The intermediate students benefitted more from glossing than beginner and advanced learners. The superiority of gloss effects for intermediate students over beginners could be explained by the possibility that students with higher proficiency levels were able to utilize glosses more effectively. The smaller glossing effect for advanced learners than intermediate learners might be due to inferencing ability. Advanced learners may have had better ability to guess the meanings of unknown words while reading, reducing their need to focus on the glosses. Given that consulting glosses directs learners’ attention away from reading, advanced learners might have more frequently ignored glosses while inferring word meanings in context.
The participants’ institutional levels were not significant; however, there was a trend that students at language schools benefitted more from glossing than students at secondary schools. This could be explained by students’ motivational factor. Students at language schools might be more motivated to learn a target language compared to students at secondary schools or universities, and they might have paid more attention to unknown words glossed in the text while thinking that reading is for the sake of learning the language.
LIMITATIONS AND FUTURE DIRECTIONS
Because the current meta-analysis only focused on learning gains, it is not clear why specific gloss formats led to greater learning than other formats. To expand on earlier studies, it would be helpful to look into the process of learning while reading glossed materials (e.g., Rott, Reference Rott2005). For example, recent studies utilizing eye-tracking technology address learners’ behavior while reading with textual glosses, pictorial glosses, and multimodal glosses (Boers, Warren, Grimshaw et al., Reference Boers, Warren, Grimshaw and Siyanova-Chanturia2017; Warren et al., Reference Warren, Boers, Grimshaw and Siyanova-Chanturia2018). Future research looking at learners’ cognitive processes while learning using eye-tracking and/or think-aloud protocols may further reveal how learners benefit from different glossing approaches and which condition maximizes the effectiveness of glossing.
The present study revealed several areas requiring more attention. First, it would be helpful to further examine the effectiveness of interlinear glosses and audio glosses as we found few studies investigating these gloss formats. Second, regarding the interaction between gloss language and L2 proficiency, future studies should recruit advanced learners to compare the effects of L1 and L2 glosses. Studies recruiting learners of different proficiencies to investigate the relative effectiveness of different gloss languages may also allow a more direct and accurate examination. Third, clear reporting of L2 proficiency based on specific standardized references (e.g., Common European Framework of Reference for Languages or American Council on the Teaching of Foreign Languages Proficiency Guidelines) may allow future meta-analyses to more accurately investigate the relationship between treatment and proficiency. Fourth, most of the included studies focused on form-meaning mapping. Although this is common in vocabulary research (Uchihara et al., Reference Uchihara, Webb and Yanagisawa2019), exploration of depth of vocabulary knowledge (e.g., grammatical functions, collocations, and associations) may further reveal how vocabulary knowledge develops from glossed reading (Webb, Reference Webb2007; see also Yanagisawa & Webb, Reference Yanagisawa, Webb and Webb2019 for a review of various approaches to measuring depth of vocabulary knowledge). Lastly, the effects of glosses may be influenced not only by each factor (e.g., gloss formats, characteristics of learners and texts, and research methodology) but also by combinations of these factors. For example, it is difficult to treat L2 proficiency and text difficulty separately because researchers may select easier reading materials for less proficient learners and vice versa. This is something that meta-analysis cannot address. Hence, individual studies are needed to examine further complicated relationships among variables and how those influence students’ vocabulary learning.
The process of this systematic analysis also found several methodological features requiring further attention. First, we found that test reliability statistics (e.g., Cronbach’s α) on vocabulary test scores were not always reported; among the included 42 studies, only 16 studies (38.1%) reported test reliability statistics. Inconsistent reporting practice of test reliability was also observed by Uchihara et al. (Reference Uchihara, Webb and Yanagisawa2019) who found that only 24% of included studies reported this. Test reliability measures not only indicate the reliability of the test but also help authors identify idiosyncratic target items.
Second, while most of the studies (39 studies, 92.9%) recruited Foreign Language (FL) students as participants, only a few studies (3 studies, 7.1%) recruited Second Language (SL) students. Different learning contexts might influence students’ attitude, strategies while reading, and familiarity with certain approaches to glossing, all of which potentially results in different effects of glossed reading. Future research should investigate glossing effects in different contexts, especially in SL contexts.
Lastly, we found that occasionally reading materials, test formats, and directions to participants were not so clearly described in studies. This prohibits exploring other potential moderator variables such as whether glossed words were underlined/highlighted, which language was used for testing, and whether participants were explicitly told to look up unknown words while reading. Providing materials (texts, tests) in the study as an appendix may enhance the clarity of studies. Following a recent trend in the field recommending open data and open materials (e.g., Marsden, Trofimovich, & Ellis, Reference Marsden, Trofimovich and Ellis2019), we would like to encourage researchers to release their materials (e.g., texts, test formats) and results dataset making them publicly available if possible. This may make future replications easier and more vigorous, as well as enhancing the transparency of the research. Having access to open materials and datasets helps future meta-analyses to provide a clearer picture of the effects of glossed reading, as well as make more accurate and robust estimations by taking advantage of individual participant data (see e.g., Cooper & Patall, Reference Cooper and Patall2009).
SUPPLEMENTARY MATERIAL
To view supplementary material for this article, please visit https://doi.org/10.1017/S0272263119000688
