Reading comprehension requires the simultaneous operation of the underlying components of working memory (WM) storage and processing. That is, readers must maintain relevant information in storage while processing and incorporating incoming information to create a coherent understanding of a text (Just & Carpenter, Reference Just and Carpenter1992). A relationship between WM and second language (L2) reading comprehension has been shown in numerous individual studies with additional support provided by a meta-analysis of L2 reading correlates (Jeon & Yamashita, Reference Jeon and Yamashita2014) and of WM and L2 outcomes (Linck, Osthus, Koeth, & Bunting, Reference Linck, Osthus, Koeth and Bunting2014). However, concerns have been raised regarding inconclusive findings on the correlation between WM and L2 reading comprehension, which ranges from little (e.g., r = –.07, in Adams & Shahnazari-Dorchech, Reference Adams and Shahnazari-Dorcheh2014) to high and positive (e.g., r = .79, in Walter, Reference Walter2004). Alptekin, Erçetin, and Özemir, (Reference Alptekin, Erçetin and Özemir2014) among others alert researchers that the differences in the WM measures and particularly in different versions of the reading span task (RST) may be contributing to the discrepancies.
The RST is one of the most commonly used measures for examining WM capacity. The common format of a RST asks a participant to read a sentence while trying to hold the final word of the sentence in memory, to indicate semantic or grammatical acceptability of the sentence just read, and finally to report the final word of each sentence in a set when the set is finished. The sets get progressively longer, and the procedure is repeated. The RST was originally developed by Daneman and Carpenter (Reference Daneman and Carpenter1980) in their attempt to create a tool that can tap the underlying components of WM, such as storage and processing. Their seminal work on the influence of WM capacity in reading with the debut of the RST has paved the way for researchers investigating the relationship between WM and reading. Since then, the RST has become a standard way of measuring WM due to its comprehensive involvement of both components of WM, differing from simple span tasks such as digit span and word span tasks that fail to do so.
Although the format of the RST mentioned above is currently the most commonly adopted version, variations in the task design are common in research that explores the link between WM and reading comprehension. Yet, little work has been done to examine how methodological features of the RST might influence the final WM scores and its correlation with reading comprehension. Reliability and validity of the RST have also been called into question due to the lack of consistencies in task design, which has resulted in “little consensus on conceptualization, operationalization, and measurement” of WM (Alptekin & Erçetin, Reference Alptekin and Erçetin2009, p. 37). In addition, attention should be paid to assessment of reading comprehension as it is also responsible for how a study on WM and L2 reading arrives at its conclusion.
Providing a quantitative summary based on the cumulative evidence allows for systematic evaluation of methodologies and patterns in findings and thus contributes to constructive debate on methodological issues and improvements in future research (Norris & Ortega, Reference Norris, Ortega, Norris and Ortega2006). In response to the growing body of research on the role of WM in L2 reading comprehension and the lack of standardization of measurement practices, the current meta-analysis aimed to gain more insights into the variability in the study outcomes.
WM and reading comprehension
WM, a limited-capacity system that is responsible for temporary storage of information while engaging in cognitively demanding tasks (Baddeley & Hitch, Reference Baddeley, Hitch and Bower1974), has been shown to be a predictor of reading comprehension (Daneman & Carpenter, Reference Daneman and Carpenter1980; Daneman & Merikle, Reference Daneman and Merikle1996; Just & Carpenter, Reference Just and Carpenter1992; Linck et al., Reference Linck, Osthus, Koeth and Bunting2014). Reading comprehension places considerable demands on WM due to a constant model building; readers undergo the simultaneous process of extracting pieces of information and consolidating them into larger chunks (Koda, Reference Koda2005). That is, readers must connect successive words, phrases, sentences, and propositions to construct a coherent mental representation of a text (Kintsch, Reference Kintsch1998, Reference Kintsch, Alvermann, Unrau and Ruddell2013). During this process, WM holds necessary information that is currently being read simultaneously keeping it available for integration with constantly incoming information and with readers’ prior knowledge held in their long-term storage.
Further, the central executive component of WM (i.e., executive functions) plays a pivotal role in reading comprehension (Daneman & Carpenter, Reference Daneman and Carpenter1980). As a cognitive control mechanism, the central executive is responsible for the control and regulation of cognitive processes, such as directing and allocating attentional resources (Baddeley, Reference Baddeley1990; Miyake et al., Reference Miyake, Friedman, Emerson, Witzki, Howerter and Wager2000; Miyake & Shah, Reference Miyake and Shah1999). While the mental representation of a text is constantly being formed during reading, readers engage in high-level processes such as making inferences, integrating successively encountered ideas, updating their interpretations of the text, and modifying information in WM. Such processes demand a great level of attentional control, regulated by the central executive (Miyake et al., Reference Miyake, Friedman, Emerson, Witzki, Howerter and Wager2000; Palladino, Cornoldi, de Beni, & Pazzaglia, Reference Palladino, Cornoldi, de Beni and Pazzaglia2001). Individuals differ in their abilities to regulate the dynamics of cognition, coordinate the processing and storage functions, and allocate attentional resources while involved in a cognitively demanding task such as reading comprehension (Gathercole & Baddeley, Reference Gathercole and Baddeley1993).
After the pioneering work of Daneman and Carpenter (Reference Daneman and Carpenter1980), who developed the RST, numerous studies have explored the relationship between WM and reading in first language (L1) contexts. The consensus is that individual differences in WM are associated with success in reading (e.g., Just & Carpenter, Reference Just and Carpenter1992; Seigneuric & Ehrlich, Reference Seigneuric and Ehrlich2005; Seigneuric, Ehrlich, Oakhill, & Yuill, Reference Seigneuric, Ehrlich, Oakhill and Yuill2000; Swanson & Berninger, Reference Swanson and Berninger1995; Turner & Engle, Reference Turner and Engle1989). In a meta-analysis with 77 studies conducted by Daneman and Merikle (Reference Daneman and Merikle1996), WM’s positive association with reading comprehension was confirmed with weighted effect sizes (correlation coefficient) that ranged from .41 (global comprehension) to .52 (specific comprehension). Studies that compared skilled and less skilled readers’ performance have shown that deficits in reading comprehension are associated with a lack of support from the central executive component WM particularly because the central executive controls attentional resources (McVay & Kane, Reference McVay and Kane2012; Swanson & Berninger, Reference Swanson and Berninger1995). Moreover, even when other strong predictors of reading were controlled (e.g., vocabulary knowledge), the effect of WM held up and showed its unique contribution to reading comprehension (Cain, Oakhill, & Bryant, Reference Cain, Oakhill and Bryant2004; Seigneuric et al., Reference Seigneuric, Ehrlich, Oakhill and Yuill2000).
Similar results have been found in L2 reading comprehension, substantiating the role of WM in reading regardless of learning contexts. Alptekin and Erçetin (Reference Alptekin and Erçetin2010, Reference Alptekin and Erçetin2011) have explored this domain in series of studies and consistently reported a positive correlation between WM and L2 passage-level comprehension ranging from .40 to .45, which are medium-sized effects (see Plonsky & Oswald, Reference Plonsky and Oswald2014). In a meta-analysis of correlates of reading, Jeon and Yamashita (Reference Jeon and Yamashita2014) investigated individual ability differences, including WM, which influence L2 passage-level reading comprehension. Although classified as a “low-evidence correlate” due to only ten studies being included in the analysis, WM did show a correlation with L2 reading (r = .42). Linck et al. (Reference Linck, Osthus, Koeth and Bunting2014) conducted a meta-analysis with 79 studies on L2 outcomes, subdivided into L2 comprehension and production, and further established that WM serves as a cognitive component underlying L2 processing. Both L2 production and comprehension were significantly associated with WM (r = .27 and .24, respectively), which supports Dehn’s (Reference Dehn2008) claim that the extent of language learning is influenced by the capacity and effective functioning of WM.
Linck et al. (Reference Linck, Osthus, Koeth and Bunting2014) have undoubtedly provided a comprehensive quantitative analysis of the role of WM in L2 learning in terms of production and comprehension. However, WM’s specific association with passage-level reading comprehension, as targeted by Jeon and Yamashita (Reference Jeon and Yamashita2014), is masked due to how language comprehension was defined in Linck et al.’s meta-analysis, which includes listening, morphosyntactic processing, and sentence processing. Therefore, additional light can be cast upon the specific link between WM and L2 reading comprehension by zooming in on studies that addressed more precise questions with respect to passage-level reading comprehension, that is, global reading comprehension skills as generally understood in language learning contexts, to expand on Jeon and Yamashita’s work.
In addition, methodological attention should be paid to WM assessment procedures, which tend to be inconsistent across studies. One of the advantages of meta-analyses and research syntheses is being able to compare and contrast findings across studies and identify gaps in research methodologies (Norris & Ortega, Reference Norris, Ortega, Norris and Ortega2006). As a case in point, Linck et al. (Reference Linck, Osthus, Koeth and Bunting2014) addressed the issue of inconsistent designs of WM measures by examining potential moderating influences of some features such as task language (L1 vs. L2) and complexity (simple vs. complex). The distinction between simple and complex tasks refers to the fact that simple span tasks (e.g., digit span) primarily tap storage whereas complex tasks (e.g., RST) involve both storage and processing. Linck et al.’s results paralleled Daneman and Merikle’s (Reference Daneman and Merikle1996) finding that WM assessed through complex measures that combine storage and processing is a stronger predictor of reading than simple measures that exclusively rely on storage. However, variations even within the same type of WM task have been pointed out (Juffs & Harrington, Reference Juffs and Harrington2011; Wen, Reference Wen2012), which warrants further scrutiny of the potential effects of the methodological features of the task on the study outcomes reported thus far.
Reading span task as a WM measure
Since Daneman and Merikle’s (Reference Daneman and Merikle1996) meta-analysis, WM tasks designed based on a dual-task paradigm to tap both processing and storage components, such as the RST, have been preferred. The RST is almost synonymous with the WM test (Wen, Mota, & McNeill, Reference Wen, Mota and McNeil2015). In the original RST (Daneman & Carpenter, Reference Daneman and Carpenter1980), participants were asked to read aloud a series of sentences and to recall the final word of each sentence included in the set, which became progressively longer. The set represented a span size, which ranged from 2 to 6. The idea was that participants must concurrently store and process information, which is at the core of the construct of WM. However, this original task received criticism for not directly assessing the processing component because there could be trade-offs between processing and storage components; participants might be sacrificing processing information to focus more on accurately recalling final words. If this does happen, scores based solely on storage will not accurately reflect one’s processing that is an equally important component of WM, nor will the scores reflect participants’ efficiency in carrying out the task (Leeser, Reference Leeser2007; Walter, Reference Walter2004). Thus, although their task “requires both processing and storage, it only measures storage” (Waters & Caplan, Reference Waters and Caplan1996, p. 73).
To address this issue, Waters and Caplan (Reference Waters and Caplan1996) argued that participants’ total level of performance can be accurately captured only when three components of the task—storage, processing, and speed—are combined. Thus, they added a secondary task to tap processing more explicitly by requiring participants to respond to a semantic judgment task after reading a sentence and measured reaction times for completing the judgment task to take processing speed into account. Scores from all three submeasures were transformed into z scores and were averaged to represent one’s WM capacity. According to Conway and colleagues (Reference Conway, Kane, Bunting, Hambrick, Wilhelm and Engle2005), this scoring system is better at discriminating between individuals and shows higher reliability. In support of this claim, Carretti, Borella, Cornoldi, and de Beni (Reference Carretti, Borella, Cornoldi and de Beni2009) have found that a measure that includes a task that taps attentional control better reflects one’s WM capacity. They also added that requiring verbal processing in the task as opposed to arithmetic processing (e.g., operation span task) can better distinguish between skilled and less skilled readers.
Waters and Caplan’s (Reference Waters and Caplan1996) scoring system was later adopted by a few studies (e.g., Leeser, Reference Leeser2007; Shin, Dronjic, & Park, Reference Shin, Dronjic and Park2019). However, a more common system is the one that includes a storage task and a processing task without taking reaction times into account (e.g., Chun & Payne, Reference Chun and Payne2004; Joh, Reference Joh2015; Kozan, Erçetin, & Richardson, Reference Kozan, Erçetin and Richardson2015; Osamu, Reference Osamu2006), and quite a number of studies still used the original method that relies only on the storage component (e.g., Bailer, Reference Bailer2011; Chun & Payne, Reference Chun and Payne2004; Shibasaki, Tokimoto, Ono, Inuoe, & Tamaoka, Reference Shibasaki, Tokimoto, Ono, Inoue and Tamaoka2015). Although considerable research has been conducted on the role of WM in reading, inconsistencies in WM assessment procedures make it difficult to accurately compare findings across studies. Because measurement of a construct is what essentially shapes our understanding of the construct, this issue becomes a limitation in the domain of WM as it relates to L2 reading as well as in other domains of L2 research (Wen, Reference Wen2012).
Potential effects of task types and methodological features on study outcomes
The correlation between WM and L2 reading comprehension may also vary as a function of other features of the RST. For instance, some studies used L2 sentences in the RST for comparability and consistent modality purposes (e.g., Alptekin & Erçetin, Reference Alptekin and Erçetin2009, Reference Alptekin and Erçetin2011; Harrington & Sawyer, Reference Harrington and Sawyer1992; Zhang, Reference Zhang2015) while others used L1 sentences to avoid L2 proficiency confounding the result (e.g., Adams & Shahnazari-Dorcheh, Reference Adams and Shahnazari-Dorcheh2014; Bailer, Reference Bailer2011; Choi, Reference Choi2013). Results regarding WM capacity in the L1 and L2 have been mixed; some have found that the storage capacity in the L1 was bigger than the L2 and that WM and language proficiency interact (e.g., van den Noort, Bosch, & Hugdahl, Reference van den Noort, Bosch and Hugdahl2006; Walter, Reference Walter2004), while others found little difference between L1 and L2 WM (Alptekin & Erçetin, Reference Alptekin and Erçetin2010). However, Osamu (Reference Osamu2006) and Alptekin and Erçetin (Reference Alptekin and Erçetin2010) reported that L1 WM did not correlate with L2 reading comprehension while L2 WM did, suggesting that L2 reading comprehension may rely on an L2-specific WM resource pool.
In addition, there are a few other methodological features, though not as frequently explored, that may contribute to the heterogeneous findings of previous research. These include the final word recall order in the storage task, processing task type, and length of sentences in the RST. Some studies ask participants to recall the final words in the same order they were presented in the set (e.g., Adams & Shahnazari-Dorcheh, Reference Adams and Shahnazari-Dorcheh2014; Chun & Payne, Reference Chun and Payne2004) while others do not impose any restrictions (e.g., Joh, Reference Joh2015; Leeser, Reference Leeser2007). Some studies have used a semantic judgment task to assess the processing function of WM, in which participants must decide if the sentence is semantically plausible (e.g., Shin et al., Reference Shin, Dronjic and Park2019; Tsai, Reference Tsai2014) whereas other studies used a grammatical judgment task (e.g., Alptekin & Erçetin, Reference Alptekin and Erçetin2010; Bailer, Reference Bailer2011). In addition, studies tend to differ in terms of the length of sentences included in the RST, which can be found as short as 5 words (Zhang, Reference Zhang2015) and as long as 18 words (Choi, Reference Choi2013).
Due to the relatively young history of exploration of individual differences in WM in the context of L2 reading, research in this domain is still evolving especially from a methodological standpoint. Researchers have addressed the inconsistent use of WM measures (Conway et al., Reference Conway, Kane, Bunting, Hambrick, Wilhelm and Engle2005; Juffs & Harrington, Reference Juffs and Harrington2011) but on a rather large scale without looking specifically into variations in WM tasks and their features. Alptekin et al. (Reference Alptekin, Erçetin and Özemir2014) have acknowledged that there exists little consensus on operationalization and measurement of WM and investigated whether different designs of the RST would make a difference in study outcomes. However, the focus was only on a couple of features (e.g., task language and processing task), leaving some methodological issues unresolved. Given the discrepancies among studies in relation to the features of the RST, the domain may benefit from more synthetic efforts with a collective approach that can produce synthesized effect estimates. In other words, methodological features of the RST may be moderator variables, variables that influence the strength of the relationship between the primary variables of interest (Hedges & Pigott, Reference Hedges and Pigott2004), which, in this case, are WM and L2 reading comprehension.
It should be noted that passage-level reading comprehension measures exhibit different features across studies as well. As discussed above, the process of reading compression requires some of the same abilities embodied by WM, such as integrating pieces of information to make inferences and updating the information held in memory accordingly. In this meta-analysis, reading comprehension constitutes half of the equation in the correlations in question. For these reasons, how reading comprehension is assessed with respect to the type of materials and the format of the test can have an impact on how studies arrive at the relationship between WM and L2 reading comprehension. Examples of features of a reading comprehension test that may moderate study outcomes include the number of texts, text type (e.g., narrative and expository), and item type (e.g., multiple choice questions). The strength of the relationship between WM and L2 reading comprehension can be sensitive to these features in addition to those of the WM measure addressed above. Thus, a systematic coding of these methodological features can lead to identifying the relationships among study findings and measurement procedures that may have been obscured in the primary studies (Lipsey & Wilson, Reference Lipsey and Wilson2001).
The present meta-analysis
The purpose of this meta-analysis is two-fold. First, with the advantage of meta-analysis that provides a method for quantitatively expressing a unified picture (Cooper, Reference Cooper2016), the study set out to obtain the magnitude of the overall relationship between WM and L2 reading comprehension by aggregating correlations reported in primary studies. More specifically, this study aimed to add to the findings of Jeon and Yamashita (Reference Jeon and Yamashita2014) and Linck et al. (Reference Linck, Osthus, Koeth and Bunting2014) by providing a quantitative description of the relationship between WM and L2 passage-level reading comprehension. Second, this meta-analysis examines whether or not methodological features of the tasks affect the strength of the relationship between WM and L2 reading comprehension. That is, moderator analyses were carried out to examine the interaction of the relationship and moderator variables and to identify (moderator) variables that help explain variability in the relationship (Hedges & Pigott, Reference Hedges and Pigott2004). Through the moderator analyses, the study intended to examine the measurement practices as a means to offer suggestions for future studies to ensure more consistent administration and assessment of the tasks. The following research questions (RQs) are addressed in this endeavor:
-
RQ1: What is the strength of the overall relationship between L2 reading comprehension and WM measured through the RST?
-
RQ2: To what extent does the relationship between L2 reading comprehension and WM vary as a function of methodological features of the WM and L2 reading comprehension tasks?
Method
Literature Search
Efforts were made to conduct a careful and exhaustive literature search with the goal of mitigating publication bias and providing as comprehensive a picture of current research as possible. The steps are summarized in Figure 1. Relevant studies were located in major databases (Academic Search complete, Dissertation Abstracts, ERIC, EBSCOhost, PsycINFO, and Linguistics and Language Behavior Abstracts) as well as search engines (Google Scholar and Google) by using various combinations of the following key terms: working memory, reading span, reading, reading comprehension, comprehen*, reading ability, L2, second language, foreign language, and bilingual* (see Appendix A). The websites of peer-reviewed journals with L2 and psycholinguistics research orientation (Applied Psycholinguistics, Journal of Psycholinguistic Research, Second Language Research, and Studies in Second Language Acquisition) were inspected using the same key words to find relevant articles in case of missing articles in the selection. Conference programs of the American Association for Applied Linguistics from 1998 to 2019 were also searched for the same purpose using key words such as working memory, RST, and reading comprehension. References in primary studies and meta-analyses on similar topics (Jeon & Yamashita, Reference Jeon and Yamashita2014; Linck et al., Reference Linck, Osthus, Koeth and Bunting2014) were consulted and used as a starting point for forward and backward chain searches to ensure a complete study selection.
Flow of information through the different phases.
Inclusion criteria
Following the search phase, a set of inclusion criteria was created to accommodate the focus of the analysis:
-
1. The study identified its participants as L2 learners with no known learning disabilities or language impairments.
-
2. The study assessed L2 passage-level reading comprehension.
-
3. The study included RST as at least one of the measures of WM.
-
4. The study reported a correlation between L2 passage-level reading comprehension scores and WM composite scores or statistical information necessary to calculate or transform into a correlation coefficient (e.g., t and F).
Exclusion criteria
-
1. The study is a duplicate of a study that is already included in the selection.
-
2. The study operationalized L2 reading as sentence processing or grammatical judgment as opposed to passage-level comprehension.
-
3. The study did not report critical information for the current meta-analysis (e.g., correlation between WM and L2 reading comprehension).
A small number of studies included more than one sample, but because the subset of participants were independent (e.g., Adams & Shahnazari-Dorchech, Reference Adams and Shahnazari-Dorcheh2014; Lee, Reference Lee2014) or because different types of task were administered to the participants to assess the constructs of interest (e.g., Alptekin & Erçetin, Reference Alptekin and Erçetin2009; Sagarra, Reference Sagarra2017), each sample in the studies was included separately (see Plonsky & Oswald, Reference Plonsky and Oswald2014, for recommended practice for data dependencies). This inclusive approach can also be supported by the fact that the focus of this current meta-analysis is to investigate whether or not different task types moderate the correlation between WM and L2 reading comprehension. Attempts were made to contact the authors of studies that met all criteria with the exception of Inclusion Criterion 4 and that reported separate subscores of WM but not the composite scores. No response was returned, resulting in 26 studies in the final study selection. Characteristics of the individual study samples are included in Appendix B.
Coding
To extract information from studies in a systematic and reliable manner, a coding scheme must be carefully developed. Two distinct parts of the coding protocol, according to Lipsey and Wilson (Reference Lipsey and Wilson2001), are study descriptors and the findings of the study, serving as independent and dependent variables, respectively. To answer RQ 1, the correlations between WM and reading comprehension were recorded. To answer RQ 2, the study characteristics part of the coding scheme was further broken down to (a) sample characteristics, (b) methodological features of RST, and (c) methodological features of a reading comprehension test. The effect sizes (r) recorded for RQ1 served as the dependent variable. The methodological features of the RST and reading comprehension test served as the independent variables. Once these superordinate categories in the coding scheme were determined, subordinate ones were identified by consulting the literature for sources of variability in the relationship between WM and L2 reading comprehension, such as task language and presence/absence of a processing task (see Table 1).
Variables and their definitions/values in the coding scheme
There were a few occasions in which adjustments were made when recording effect sizes. The majority of the primary studies reported a correlation between WM composite scores and L2 reading comprehension scores. However, there were a couple of studies that adopted different research design and reported an estimate of variance in L2 reading accounted for by WM. In these cases, the reported η2 value was transformed into r by taking the square root (η2 is calculated the same way as r squared), or descriptive statistics and other information were used to generate r by using the effect size conversion spreadsheet created by Daniel Lakens (https://osf.io/fm4gc/; see Lakens, Reference Lakens2013, for details).1
Following Plonsky and Oswald’s (Reference Plonsky, Oswald, Mackey and Gass2012) suggestions, more variables rather than fewer were coded for to take a more inclusive approach. In particular, surveying the primary studies in the selection revealed several methodological features that varied across studies, which hold potential for influencing study outcomes. The coding scheme underwent two revisions after consulting experts on methodological and substantive matters. To ensure reliability and accuracy of the coding, a second coder was recruited and trained by holding a norming session in which the researcher and second coder coded three studies together. After the norming session, half of the samples in the study selection were coded by a second coder with high interrater reliability (κ = .95). As low-inference codes were made in the current meta-analysis, no subjective judgments were necessary. At any occurrence of discrepancies, however, the corresponding articles were consulted again to resolve the issue.
Analysis
Although a correlation coefficient between two continuous variables serves as an index of an effect size, most meta-analyses of correlations transform these values into their associated z scores for combining effect size estimates and testing for moderators because correlations are not normally distributed (Cooper, Reference Cooper2016). After the analysis is performed on the transformed values, however, it is common to convert the values back to correlations for presenting the results, which also facilitates interpretation (Borenstein, Hedges, Higgins, & Rothstein, Reference Borenstein, Hedges, Higgins and Rothstein2009). The current meta-analysis followed this procedure during which it was found that r values and the corresponding z scores were almost the same as each other. Nonetheless, r values were transformed to z scores first for analyses, which were converted back to r for presenting the results.
The analysis was conducted using the metafor package (Version 1.9-5; Viechtbauer, Reference Viechtbauer2010) in R (R Core Team, 2015). In order to address RQ1, concerning the overall relationship between WM and reading comprehension, the sample-level correlations from each study were averaged. In doing so, the effect sizes (r) were weighted by their corresponding sample size to assign more weight to more precise studies that carry more information to present a reflection appropriate for the statistical power (Borenstein et al., Reference Borenstein, Hedges, Higgins and Rothstein2009).
RQ2 addressed variability in observed correlations as a function of the tasks used to measure L2 reading comprehension and WM. To answer RQ2, each sample was coded for methodological features of both the reading comprehension test and RST. Most of the variables were coded dichotomously (e.g., L1 vs. L2; Presence vs. Absence), and the mean effect sizes for subgroups based on this coding were then compared in moderator analyses. However, there were a few continuous variables (e.g., length of sentences) whose numerical values were recorded in the coding sheet. To examine their moderating effects on effect sizes, correlation analysis was performed.
Results
Checking for outliers
Because providing a reasonable summary of quantitative findings of research is the main purpose of meta-analysis, outliers with disproportionate influence on the aggregated results can potentially distort the finding (Lipsey & Wilson, Reference Lipsey and Wilson2001). Upon examining the distribution of effect sizes, one independent sample from a study (Walter, Reference Walter2004) was found to have produced an extreme value (r = .79, z = 1.07; see Figure 2). It was also over 3 SD from the mean of the effect sizes and thus was removed as advised in Hedges and Olkin (Reference Hedges and Olkin1985). The results provided below are based on 25 studies and 37 unique samples.
Boxplot of the distribution of effect sizes.
Checking for publication bias
Publication bias as a threat to validity is a commonly discussed issue in meta-analysis due to the possibility of drawing a false conclusion if based on an unrepresentative body of research (Rothstein, Sutton, & Borenstein, Reference Rothstein, Sutton and Borenstein2005). Though efforts were made to conduct an inclusive and exhaustive search, a funnel plot was constructed and examined in order to assess the presence of publication bias. As illustrated in Figure 3, the studies are distributed around the mean effect size almost symmetrically though not perfectly, indicating the likely absence of publication bias. A regression test for funnel plot asymmetry also confirmed the symmetry (z = .97, p = .33).
Funnel plot of effect sizes in the z score form and standard error.
RQ1: Overall relationship between L2 reading comprehension and WM
Using 25 primary studies and 37 unique samples comprising 2,682 individual participants, RQ 1 was addressed by aggregating the effect sizes (r) reported in primary studies on the relationship between WM and L2 passage-level reading comprehension. As summarized in Table 2, the overall weighted mean was .30 and very close to the unweighted mean (r = .31). Both means indicated that there is a medium association between WM and L2 reading comprehension. A test of homogeneity of effect size was performed to test if the effect sizes differed due to chance or sampling error. The Q statistic was statistically significant (Q = 70.50, p < .001), indicating systematic variations among effect sizes and thus presence of moderators. I 2 statistic suggested that 49% of the variation was due to heterogeneity rather than chance.
Overall relationship between working memory and L2 reading comprehension
Note: aUnique samples (25 primary studies). bIndividual participants. p* < .001.
RQ2: Methodological features of WM and L2 reading comprehension tasks as moderators
The magnitude of the overall relationship between WM and L2 reading comprehension served as the anchor against which the results of moderator analyses (RQ2) can be compared. The current meta-analysis is interested in not only reporting on the overall relationship but also accounting for systematic differences in the size of the outcome. It is particularly concerned with variation in effect sizes possibly being moderated by different types and features of tasks used to measure WM and L2 passage-level reading comprehension. The results of the moderator analyses are organized into two parts to present (a) RST characteristics and (b) reading comprehension test characteristics separately.
To address RQ 2, effect sizes (r) were organized in accordance with methodological features of the RST and reading comprehension tests. Some of these features have been discussed in the literature (e.g., task language, presence/absence of a processing task, scoring system, and text type) and others may not have been extensively investigated but vary across studies (e.g., final word recall order and length of sentences). As previously mentioned, the effect sizes and their corresponding z scores were very similar to each other. The author compared the results using z scores and r values and confirmed that the summary effects were almost the same. Therefore, the author presents the results using r values for ease of interpretation. The weighted results (by sample size) are summarized in Table 3 in which the mean effect sizes for each subcategory (M[r]), the number of participants included (N), the number of studies (K), standard errors (SE), and confidence intervals are reported. Nonoverlapping 95% confidence intervals indicate a statistically significant difference (at the .05 level).
Methodological features of reading span task and reading comprehension test as moderators
Note: Unreported information is not included. aBecause only one study used a picture-passage matching task, it was not included.
RST features as moderators
A total of eight methodological features were examined: task language, final word recall order, processing task, processing task type, scoring system, reporting of reliability, the number of sentences, and the length of sentences. Results regarding the first six features of the task are presented in Figure 4 as they were categorically coded. Statistically significant differences are shown with non-overlapping 95% confidence intervals. References are made to Table 3 to address the results regarding the last two features (i.e., number and length of sentences).
Effect sizes based on methodological features of the reading span task (RST). SJ, semantic judgment. GJ, grammatical judgment. S, storage only. P, processing, R, reaction times. Error bars represent 95% confidence intervals.
Task language
Task language refers to the language of sentences that was used in the RST (L1 vs. L2). The effect size was larger when L2 was used (r = .35) and closer to the overall effect size (r = .30) than when L1 was used (r = .17). That is, task language moderated the study outcomes; studies that used an L2 RST found a stronger relationship between WM and L2 reading comprehension.
Final word recall order
Final word recall order refers to the order in which final words of sentences were reported (correct vs. random). As seen in Table 3, 32 out of 37 samples employed a final word recall task to measure the storage component of the RST, whereas only five used different tasks. Thus, the moderator analyses focused on the final word recall task and how it was carried out. The effect size was larger when the final words were recalled in correct order as they were presented in the task (r = .33) than when recalled in random order (r = .23).
Processing task and type
The processing task was coded for its presence and absence in the RST. As noted previously, WM assessment varies across studies in terms of which WM components (storage and processing) are measured and how. Although storage is always measured in the RST in one form or another, that is not the case for the other WM component: processing. For this reason, whether or not a processing task was included in the RST, regardless of the task type, was examined. Presence of a processing task in the RST yielded a larger effect size (r = .31) than when WM was measured without it (r = .15). The effect size produced in the presence of a processing task was also very close to the overall mean effect size (r = .30), indicating the importance of including a processing task in the RST. However, the processing task type did not make any difference, whether it was a semantic judgment task (r = .31) or a grammatical judgment task (r = .31).
Scoring system
As for the scoring system, three patterns emerged while coding: scores were represented by (a) storage only, (b) storage and processing, or (c) storage, processing, and reaction times (processing speed). The largest effect size was found when both storage and processing scores were included in the scoring system (r = .33). When reaction times were added in the scoring method to account for processing speed, the effect size became slightly smaller (r = .30) but was exactly the same as the overall effect size (r = .30). When only storage was included, the effect size was the smallest (r = .20) and farthest from the overall effect size.
Reporting of reliability
Only 9 out of 37 samples reported reliability of the RST. The effect size was larger when the reliability was reported (r = .30) than when it was not (r = .25). This indicates that studies that reported reliability of the RST tended to find a stronger relationship between WM and L2 reading comprehension.
Number and length of sentences
In addition to the categorical moderator variables discussed thus far, also included in the analyses were two continuous variables that could potentially moderate the relationship between WM and L2 reading comprehension. As shown in Table 3, correlation analyses revealed that the number of sentences in the RST did not have an effect (r = .05). The length of sentences included in the RST and effect sizes were negatively associated (r = –.53), indicating that the longer the sentences, the smaller the effect size.
L2 Reading comprehension test features as moderators
The second part of the moderator analyses concerns the methodological features of L2 reading comprehension tests. A total of seven features were examined: text type, test type, item type, reporting of reliability, the number of texts, the number of items, and the length of texts (see Figure 5 and Table 3).
Effect sizes based on methodological features of the reading comprehension tests. Multiple choice Qs represents multiple-choice questions. Error bars represent 95% confidence intervals.
Text types
In terms of text types (narrative vs. expository), the effect size was larger when reading performance was tested on narrative texts (r = .33) than expository texts (r = .27). This result indicates that there was a stronger relationship between WM and narrative text comprehension than between WM and expository text comprehension.
Test types
Upon examining test types, two types emerged: standardized tests (e.g., TOEFL) and researcher-developed tests. Using a standardized test produced a larger effect size (r = .28) than did researcher-developed tests (r = .24), showing that a stronger relationship between WM and L2 reading comprehension was found when a standardized test was used.
Item type
Another moderator variable that influenced the study outcomes was the item type (multiple choice questions vs. written recall). Although multiple choice questions in a reading comprehension test were a popular choice for item type in the studies (k = 24), the effect size was larger when written recall was used (r = .35) than when multiple choice questions were used (r = .23). This result shows that the association between WM and L2 reading comprehension was stronger when reading performance was assessed in the form of written recall.
Reporting of reliability
Whether or not the reliability of the reading comprehension test was reported did not moderate the study outcomes. However, it should be noted that 25 out 37 samples did not report the reliability, which reveals the reliability reporting practices in this research domain.
Number and length of texts and number of items
The effect sizes were positively correlated with the number of texts (r = .53) and number of items (r = .46), suggesting that a stronger relationship between WM and L2 reading comprehension was found when more texts and items were included in the test. The length of texts in the reading comprehension tests, however, did not show evidence of moderating the study outcomes (r = .10).
Discussion
The overall results of the present study confirmed the medium-sized relationship between WM and L2 reading comprehension (r = .30). The overall effect size also indicates that about 9% of the variance in reading comprehension scores can be accounted for by WM capacity. The remaining 90% or so would then be attributed to additional variables such as vocabulary knowledge (e.g., Proctor, Carlo, August, & Snow, Reference Proctor, Carlo, August and Snow2005; Shiotsu, Reference Shiotsu2010), grammar knowledge (e.g., Choi, Kim, & Boo, Reference Choi, Kim and Boo2003; Shiostsu & Weir, Reference Shiotsu and Weir2007), and decoding (e.g., Nassaji & Geva, Reference Nassaji and Geva1999; Pasquarella, Chen, Lam, Luo & Ramirez, Reference Pasquarella, Chen, Lam, Luo and Ramirez2011).
The effect size found in this study is similar to what was reported in Linck et al.’s (Reference Linck, Osthus, Koeth and Bunting2014) meta-analysis (r = .26), which focused on the relationship between WM (regardless of tasks used) and language comprehension (not limited to reading comprehension). Although these effect sizes are smaller than what Jeon and Yamashita (Reference Jeon and Yamashita2014) reported (r = .42), only 10 studies on WM were included in Jeon and Yamashita’s analysis. In addition, many of the primary studies included in Jeon and Yamashita did not measure WM via the RST, which suggests that different measures of WM (e.g., digit span task, operation span task, and RST) may influence the extent to which WM is associated with L2 reading comprehension. Because the current meta-analysis focused on the studies that used the RST as a WM measure, exploring this issue is beyond the scope of the study. However, it would be a worthwhile effort to investigate what subcomponents of WM (e.g., storage or processing) are tapped by different WM measures and how they are linked to L2 reading comprehension, especially with the increasing interest in measurements and conceptualization of the construct of WM.
The second set of results provide clues as to why different effect sizes were observed across studies. In the following discussion, methodological features of the RST are discussed first, followed by the features of L2 reading comprehension tests. As previously stated, the result from RQ1—the average effect size—serves as the anchor against which the moderating effects of methodological features can be compared. Task language (L1 vs. L2) was one of the moderating variables of the study outcomes. The effect size produced through the L1 RST was smaller than the overall average effect size. Task language of the RST has been one of the sources of controversy (Erçetin, Reference Erçetin2015). On one side of the controversy, performance on L1 and L2 RSTs have been shown to be strongly correlated (Alptekin & Erçetin, Reference Alptekin and Erçetin2010; Osaka & Osaka, Reference Osaka and Osaka1992; van den Noort et al., Reference van den Noort, Bosch and Hugdahl2006). Osaka and Osaka (Reference Osaka and Osaka1992) demonstrated that WM for reading is independent of language structure. However, studies have consistently shown that L2 WM correlates with and contributes to L2 reading comprehension whereas the results have not been consistent with L1 WM (e.g., Harrington & Sawyer, Reference Harrington and Sawyer1992; Joh, Reference Joh2015), leading to the conclusion that L1 and L2 WM measures may not be completely parallel (Geva & Ryan, Reference Geva and Ryan1993).
In this respect, the current meta-analysis provides another piece of evidence that the WM outcome and its relationship with L2 reading comprehension by extension are likely to be sensitive to task language. This finding is also in line with Linck et al.’s (Reference Linck, Osthus, Koeth and Bunting2014) claim that larger correlations between WM and L2 outcomes may be found when WM is assessed through a task using the L2. Further support can be found in Miyake and Friedman (Reference Miyake, Friedman, Healy and Bourne1998) who showed in their path analysis that L1 WM is directly related to L2 WM but is indirectly related to L2 syntactic comprehension. Considering the same modality in operation, it can be suggested that L2 WM has a stronger link with L2 reading comprehension (Erçetin, Reference Erçetin2015). Examining the interaction between the two potential moderators—task language and L2 proficiency—in the present meta-analysis could have addressed the issue of L2 proficiency being a confounding factor (Linck et al., Reference Linck, Osthus, Koeth and Bunting2014). As Lipsey (Reference Lipsey2003) explains, however, disentangling the relationship between moderators to identify their unique contributions to effect sizes requires a sufficiently large number of studies, which unfortunately was not the case for the present meta-analysis.
The scoring method is another issue often discussed in relation to reliability and validity of the RST (Erçetin, Reference Erçetin2015). As shown above, the three different scoring systems adopted by studies in the sample led to three different effect sizes. It has been found that a WM measure that combines the storage and processing components is a better predictor of L2 reading (Daneman & Merikle, Reference Daneman and Merikle1996; Waters & Caplan, Reference Waters and Caplan1996). Therefore, it comes as no surprise that the presence of a processing task in addition to the final word recall task measuring storage yielded a larger effect size regardless of the type of the processing task (i.e., semantic and grammatical judgment task). As the shared resources hypothesis posits, there is a competition between processing and storage for resources from the shared pool. In order to mitigate this trade-off relationship, a composite score based on both storage and processing subscores can represent WM capacity more accurately (Leeser, Reference Leeser2007; Waters & Caplan, Reference Waters and Caplan1996). Alptekin and Erçetin (Reference Alptekin and Erçetin2009) also asserted that WM capacity can better predict L2 reading comprehension when both storage and processing are reflected in the composite scores. However, it remains unclear as to why including reaction times to account for processing speed produces a smaller effect size than when it is not included in the scoring procedures.
One explanation for this may be that there is a trade-off not only between storage and processing but also between processing accuracy and processing speed (i.e., accuracy–speed trade-off). Processing speed may be sacrificed to obtain more accurate processing. Leeser (Reference Leeser2007) reported that reaction times were negatively correlated with semantic judgment accuracy (r = –.53) and final word recall scores (r = –.26), indicating that participants may have sacrificed the speed in their efforts to recall more words and make more accurate judgments. It should be noted that taking processing speed into account is a relatively new approach in WM assessment and that there are only six studies (eight samples) in the sample of the present meta-analysis that adopted this approach. Thus, the role of processing speed should be further explored with more empirical work. Nonetheless, this finding warns against using WM tasks that do not include a processing-accuracy task (e.g., semantic judgment and grammatical judgment), let alone processing speed, which ultimately concerns reliability of the results and the operational construct of WM.
Another noteworthy finding is that the order of final word recall (correct order vs. random order) contributed to the systematic variability among studies. Little attention has been paid to the effect of the order of final word recall. However, it may be that recalling verbal items in the same order that they were presented is associated with the storage component of WM more strongly and perhaps introduces less noise; in turn, it may produce a larger effect size (i.e., a stronger relationship between WM and L2 reading comprehension) since storing and maintaining information is an ability shared with reading comprehension.
The number and length of sentences in the RST were also included in the moderator analyses. The number of sentences did not have any impact. However, the length of sentences showed a negative correlation with the effect sizes, suggesting that the longer the sentences in the RST, the weaker the relationship between WM and L2 reading comprehension. The discrepancies here are large too; the length of sentences across studies included in the sample ranges from 5 to 18 words. This is not to suggest that sentences in the RST should be as short as possible. Rather, the length of sentences is another feature of the task that researchers may need to pay attention to as it could add to the cognitively taxing nature of the RST and thus have an influence on the resulting relationship between WM and L2 reading comprehension. In any case, the emerging theme thus far is that there are overall inconsistencies in assessment of WM and, more important, that many of these features and choices moderate the effects. It is also worth noting that the reliability of the RST was reported in only 9 out of 37 samples. This finding and the lack of standardized procedures call for increased methodological rigor in terms of WM task design and reporting practices.
Examining the features of L2 reading comprehension tests as moderators provided insights on the relationship between WM and L2 reading comprehension. With respect to the text type, the link between WM and reading was stronger when working with narrative texts than expository texts. Narrative texts are more conducive to generating inferences and constructing a mental representation of text because interpretive mechanisms and knowledge structures are more likely to be tapped (Graesser, Singer, & Trabasso, Reference Graesser, Singer and Trabasso1994). Expository texts, in contrast, are often decontextualized and are concerned with delivering generic truths and thus fewer inferences are generated. Perhaps, L2 readers draw from their WM resources more when reading narrative passages as they tend to be less explicit and invite readers to fill more gaps to make sense of the text (DuBravac & Dalle, Reference DuBravac and Dalle2002).
In terms of the item type, a larger effect size was found with a reading comprehension test with written recall than multiple choice questions. This may be attributed to cognitive demands involved in completing a written recall task; because readers must rely on their memory to produce as many and accurate responses as possible in a recall task typically without having access to the material, their storage is likely tapped considerably more than when they respond to multiple choice items. That is not to say that written recall scores reflect L2 readers’ comprehension more accurately. What the finding shows is that more WM resources may be called upon in one type of reading task than another. Given that the text type and item type contribute to variability in the study outcomes, it can be suggested that the involvement of WM in L2 reading comprehension varies depending on the type of reading tasks at hand.
Similar to the discussion on the reliability reporting issue of the RST, only 12 out of 37 samples reported the reliability of the reading comprehension test though it did not make any difference in the study outcomes. Still, considering that more samples relied on researcher-developed tests (k = 26) than standardized ones (k = 11), reporting test reliability would help ensure credibility of the findings. For assessing written recall, in particular, it would be important to report the scoring scheme and interrater reliability to ensure the quality of the measure because subjective judgments may be involved. Further examination revealed that out of the 12 samples that used written recall to assess L2 reading comprehension, only 3 of them reported interrater reliability. Finally, the correlations between the effect sizes and the number of texts and questions included in the reading comprehension tests might be related to reliability of the tests. It is likely that reliability improves with more items in a test, which, in this case, may have led to less attenuation of the observed correlations.
Conclusion, limitations, and future directions
The current meta-analysis sought to systematically survey the literature to (a) examine the overall magnitude of the relationship between WM and L2 reading comprehension and (b) explore which methodological features of the tasks moderate this relationship. In doing so, it focused on the RST as it is the most widely used WM measure in the relevant domain of research. Limiting the scope of this meta-analysis to the RST allowed the researcher to delve into the variations of the task that exist in previous studies and to examine their moderating effects on the study outcomes. Although this can be seen as a strength in terms of ensuring greater homogeneity of the sample, it also meant discarding studies that used different WM measures (e.g., operation span task) which have their own advantages in assessing WM. As a result, a limited amount of studies was included in the selection, which may not be representative of all studies on the link between WM and L2 reading comprehension.
As an anonymous reviewer pointed out, quality assessment can be important for determining trustworthiness of the individual studies and identifying data that may possibly have been contaminated by other irrelevancies. However, because the purpose of this meta-analysis was to examine features of the WM task and L2 reading tests and their impacts on the study outcomes, the author opted to keep all studies that (a) met the inclusion criteria and (b) were designed and implemented in ways that allowed for drawing statistical inferences needed for this meta-analysis. One way to assess quality of an L2 research study would be to examine reliability of instruments. In this meta-analysis, the reporting of reliability was treated as one of the potential moderators rather than as a quality assessment criterion in order to examine the current reporting practices in this domain of research. Though interpretation of the results presented in this meta-analysis should be made with caution given that rigorous filtering did not occur, the decision to be inclusive goes hand in hand with the exploratory nature of the current meta-analysis.
Another potential limitation concerns the possibility of interactions among moderating variables, which were not examined due to the limited number of studies available within the target domain. Along the same line, and as pointed out by a reviewer, learner characteristics such as age and L2 proficiency are potentially important moderator variables. However, they were excluded in the moderator analyses due to insufficient variability in the sample and unavailable information. The lack of variability in the sample, in particular, demonstrates that research in this domain tends to focus on certain learner groups in terms of age and proficiency, which reveals a gap in the literature. Future meta-analyses as well as individual studies will, I hope, be able to address this gap and relationships among WM, L2 reading abilities, and learner characteristics.
Nonetheless, this meta-analysis portrayed a picture of the association between L2 passage-level reading comprehension and WM measured through the RST using a relatively homogeneous sample. It offers insights into the varying degrees of WM involvement in different types of reading comprehension, supporting task-specific WM. Although the attentional control aspect of WM governed by the central executive may be domain general, the processing function of WM can be constrained by task-related factors (Waters & Caplan, Reference Waters and Caplan1996). Therefore, more research should be done to improve our understanding of the extent to which cognitive resources are called upon in response to challenges in different types of reading.
In addition, the current meta-analysis provides useful findings on the issues in regard to conceptualizing and measuring WM and reveals inconsistencies in measurement practices. The scoring system of the RST has been gaining more interest recently. In particular, the role of reaction times representing processing speed needs further exploration as there may be a trade-off relationship with processing accuracy as well as storage. Empirical efforts to scrutinize the subcomponents of WM can also contribute to explaining the complex construct of WM. With respect to the inconsistent measurement practices, researchers should first be aware of the moderator variables identified in this meta-analysis and take caution in interpreting findings of previous research. For comparability purposes, future studies may benefit from replicating procedures of the RST adopted in previous studies that share the same research goals rather than implementing yet another variation of the task. In turn, this will allow for presenting a more coherent picture of the relationship between WM and L2 reading comprehension while enabling future research syntheses to address unresolved issues.
Acknowledgments
I wish to thank Luke Plonsky for his support and feedback on this meta-analysis. I also thank Jesse Conway for his helpful comments.
Appendix A. Search terms used for literature search in databases
-
1. “working memory” OR “reading span task”
-
2. “reading comprehen*” OR “reading abilit*”
-
3. “second language” OR “foreign language” OR L2 OR bilingual*
Combinations of 1, 2, and 3 were used.
Appendix B. Characteristics of studies in the sample
