Effect Size–Driven Sample-Size Planning

The sample-size planning process begins with effect sizes, whether it is classical power, driven by null hypothesis significance testing (NHST), or another process such as precision (Cumming, Reference Cumming2012; Norouzian, Reference Norouzian2020). In both of these approaches, researchers reference effect sizes from the relevant domain to plan the number of participants they require. Effect size metrics, as highlighted in the preceding text, quantify an observed sample’s experimental and correlational effects to enable generalizations to the population. Such metrics are numerous but tend to be organized under mean difference (d family), strength of association (r family), and odds ratio categories (Plonsky et al., Reference Plonsky, Sudina and Hu2021). L2 experimental meta-analyses have usually expressed experimental effect sizes in standardized mean difference metrics, the d family, despite the catalogued reports sometimes utilizing different metrics (e.g., Bryfonski & McKay, Reference Bryfonski and Mckay2019; Lin & Lin, Reference Lin and Lin2019; Vitta & Al-Hoorie, Reference Vitta and Al-Hoorie2020). As highlighted by Brysbaert (Reference Brysbaert2019), effect size metrics convert from one to another quite easily while expressing the same underlying effect. When d = 1.00, a group’s performance averaged one pooled standard deviation higher than another; when d = .50, the group was 50% of a standard deviation better.

Classical a priori NHST power analysis is perhaps the best-known method to plan sample sizes using effect sizes. Recent L2 methodological syntheses (e.g., Farsani & Babaii, Reference Farsani and Babaii2020; Lindstromberg, Reference Lindstromberg2016; Vitta & Al-Hoorie, Reference Vitta and Al-Hoorie2021) have exclusively coded for classical power given NHST’s prevalence among SLA quantitative research, despite arguments to move away from it (e.g., Cumming, Reference Cumming2012; Plonsky, Reference Plonsky and Plonsky2015). As highlighted by Nicklin and Vitta (Reference Nicklin and Vitta2021) and Brysbaert (Reference Brysbaert2019), such power analyses should reference relevant standardized effect sizes.Footnote ² The process entails the calculation of the sample size required to detect a certain effect size assuming predetermined alpha (α) and beta (β) thresholds. The α thresholds relates to Type I error (i.e., false positive probability) and is conventionally set at 5%, while β thresholds relate to Type II error (i.e., false-negative thresholds) and are typically set at 20% (Lakens et al., Reference Lakens, Adolfi, Albers, Anvari, Apps, Argamon and Zwaan2018). For fixed-effects testing (e.g., ANOVA and multiple regression), calculators such as G*Power (Faul et al. Reference Faul, Erdfelder, Lang and Buchner2007) are useful for conducting a priori power analyses but they can be limited in their ability to model interactions. However, this issue can be addressed using simulations (see Lakens & Caldwell [Reference Lakens and Caldwell2019] for details). Mixed-effects models where random effects (i.e., not a part of the experimental design) at the item and person levels likewise require a priori power to be addressed using simulations analyses (e.g., Brysbaert & Stevens, Reference Brysbaert and Stevens2018; Green & MacLeod, Reference Green2016).

Precision has recently been introduced within the SLA arena and acts as a useful exemplar of a non-NHST a priori, effect size–driven sample-size planning procedure (Norouzian, Reference Norouzian2020). This approach determines the sample size required to narrow the confidence interval (CI) for a certain effect size. Much like G*Power, Norouzian’s precision calculator is turnkey whereby the researcher selects the effect size, the parametric test, and the degree to which the CI is narrowed. According to Norouzian’s SLA-specific precision calculations, the trend of larger samples for small effects is present in a general sense. For example, Norouzian demonstrated that precision in regression modeling increased with sample size when effect size was held constant. Certain conditions, however, saw almost identical sample sizes providing adequate precision for different effect sizes (see Figure 3 in Norouzian, Reference Norouzian2020, p. 854).

Small samples that have not been planned with relevant effect sizes are problematic in relation to both small and large effects (Brysbaert, Reference Brysbaert2019). A small sample might lead toward a false negative discovery, or Type II Error. In this situation, a researcher determines that a small effect in the sample does not exist in the population when in reality it does, but the sample was too small to significantly detect it (as argued by Brysbaert within an NHST paradigm). Fraley and Vazire (Reference Fraley and Vazire2014) demonstrated that when samples are underpowered the distribution of observed sample effects is wider than the distribution in higher-powered samples. In other words, there is a higher chance of studies observing larger and smaller effects in relation to the true population effect. This can lead to the publication of large effect “flukes,” which occur when a large significant effect in a small sample is generalized to the population by the researcher when in reality it does not exist. For instance, Yang et al. (Reference Yang, Shintani, Li and Zhang2017) reported a very large effect size (d _s = 1.75)Footnote ³ in favor of a sentence writing L2 IVA condition group over an essay writing group while the median between-subject d in SLA was estimated as 0.70 (Plonsky & Oswald, Reference Plonsky and Oswald2014). Because Yang et al.’s sample comprised only 19 and 18 students in the respective sentence and essay writing subgroups from a single Chinese university, the concern of this huge observed sample effect being a fluke relative to the population emerges. When one couples this with the observed publication bias toward large and significant effects (Fanelli, Reference Fanelli2010), the need for L2 IVA and L2 experimental designs to plan their sample sizes with reference to relevant effect sizes before data collection begins is clear.

Given the problems that samples constructed without a priori size planning present, it is concerning that the process has been almost never observed when L2 methodological syntheses sought to investigate it. When reviewing L2 interaction inquiries, Plonsky and Gass (Reference Plonsky and Gass2011) observed that only three (1.72%) of 174 studies reported power in any fashion, with it being unclear if any of them employed a priori or post hoc procedures. Farsani and Babaii (Reference Farsani and Babaii2020) likewise observed 6 (2.10%) of 285 Iranian graduate theses reporting power without considering if any were a priori. In a review of quantitative reports published in Language Teaching Research from 1997 to 2015, Lindstromberg (Reference Lindstromberg2016) overtly mentioned that no study engaged in an a priori sample planning process. His null finding corresponded to Vitta and Al-Hoorie’s (Reference Vitta and Al-Hoorie2021) observation that none of the 56 experimental reports investigating L2 flipped learning, in which new content is presented to students before class as homework (see Mehring, Reference Mehring, Mehring and Leis2018 for more on the approach), featured a priori power assessments while two (3.57%) considered post hoc power considerations.

This observed, albeit limited application of post hoc (NHST) power highlights another issue with the field’s failure to conduct a priori power analyses. Post hoc power analysis, which is classical NHST power analysis conducted after data has been collected, is routinely criticized as illogical because the power serves the purpose of determining the sample size an inquiry requires (e.g., Cumming, Reference Cumming2012; Field, Reference Field2018). Proponents of post hoc power emphasize its utility in assessing the power of a finding assuming that it truly exists in the population. This utility however has been noted as “pointless and potentially misleading” (Perugini et al., Reference Perugini, Gallucci and Costantini2018, p. 3), with Cohen (Reference Cohen1988) noting that sample-size planning should occur before the study is undertaken. Another option available to researchers is sensitivity analysis, which determines the minimal effect size that a sample can significantly detect given its size. This effect size is then the baseline that the study can accept in relation to detecting effects in the sample that generalize to the population. Some researchers have presented sensitivity analysis as a compromise between the ideal a priori and problematic post hoc sample-size determination analyses (Perugini et al., Reference Perugini, Gallucci and Costantini2018). A sensitivity analysis probably does avoid the “illogical” flaws of its post hoc counterpart. Nevertheless, only a priori effect size–driven planning with relevant effect sizes located in prior research allows researchers to situate their proposed sample size in the existing body of inquiries (Brysbaert, Reference Brysbaert2019; Nicklin & Vitta, Reference Nicklin and Vitta2021).

Randomization: Sampling and Assignment

When an experimental design features a randomized sample with random assignment of participants, it has stronger representativeness in relation to the intended population (Fisher, Reference Fisher1935). “Random” and “randomized” imply that chance drives either the selection or assignment process. This removes the bias that a researcher-controlled procedure might introduce, which in turn degrades the generalizability of the sample vis-à-vis the intended population.

Randomized sampling is the process by which participants are randomly selected from the intended population. According to most frameworks (e.g., Harter, Reference Harter and Lavrakas2008), randomized sampling is the superordinate under which one can subsume probability sampling procedures such as stratification and clustering. Stratification, as defined by Harter, is the identification of subgroups, or strata, within the population that should be represented in the sample, with participants randomly selected from each strata. Clustering is the random selection of participants from randomly selected clusters, which are intact groups existing in the population (Trochim et al., Reference Trochim, Donnelly and Arora2016). Among L2 researchers, such sampling is rarely implemented, given the practical constraints that the field faces. In a recent review of L2 research, only 2 (8.4%) of the 282 reviewed samples employed randomized sampling, subsuming fully randomized, clustered, and/or stratified procedures (Farsani & Babaii, Reference Farsani and Babaii2020). In a similar vein, Vitta and Al-Hoorie (Reference Vitta and Al-Hoorie2021) informally observed that none of the 56 L2 flipped learning experimental reports that were coded involved a randomized sampling process and thus the procedure went unanalyzed in the synthesis. Earlier reviews of L2 samples omitted the issue entirely with only randomized assignment being analyzed (e.g., Plonsky, Reference Plonsky2013).

The potential of randomized sampling within L2 research is noteworthy, however. Hiver and Al-Hoorie (Reference Hiver and Al‐Hoorie2020) could not replicate the conclusions and findings of You et al.’s (Reference You, Dörnyei and Csizér2016) L2 learning motivation study. One of the differences between the studies was that the replication study employed a randomized stratified sample, while You et al. employed a nonprobability snowball sample. Snowball, or chain sampling, involves participants enlisted for earlier iterations of the research assisting in participant recruitment for later iterations (see methods in You et al.). According to Trochim et al. (Reference Trochim, Donnelly and Arora2016), this procedure is subject to low external validity, entailing biased results and a lack of generalizability. While Hiver and Al-Hoorie’s failure to replicate cannot be fully ascribed to the sampling difference, this finding invites randomized sampling replications of prior findings from early studies that employed convenience samples.

Unlike with randomized sampling, L2 methodological syntheses have reported that L2 researchers are more prone to employ randomly assigned experimental conditions. Random assignment is the process of assigning participants to experimental conditions (e.g., treatment and comparison) by chance. Farsani and Babaii (Reference Farsani and Babaii2020) observed that 100 (58%) of 170 classroom experiments featured randomized assignment at either the class or participant level. This was higher than the overall 47% observed by Plonsky (Reference Plonsky2013) seven years’ earlier in 606 quantitative reports published in popular L2 SSCI (Web of Science’s Social Science Citation Index) journals, and also higher than the 32% observed by Lindstromberg (Reference Lindstromberg2016). The relatively high observed frequency of random assignment in comparison with random sampling and sample-size determination procedures allows for assessment of the association between time and random assignment. However, divergent findings have been reported. Lindstromberg observed a small correlation (ρ = .17) between year of publication and the use of random assignment in 76 between-subject comparison experimental designs. Plonsky (Reference Plonsky2014), however, observed that the use of random assignment in SSCI journals’ reports decreased when comparing the 1990s (40%) to the 2000s (36%).

While random assignment at the participant level is the gold standard (see Rogers & Révész, Reference Rogers, Révész, McKinley and Rose2020) and corresponds to Fisher’s (Reference Fisher1935) guidance, L2-centric reviews have made the distinction between random assignment at either the class/group or participant level. In classroom-centric SLA subfields such as L2 IVA, assessing random assignment at the class/group level is sensible as research is often conducted on intact classes. For instance, Peters (Reference Peters2019) randomly assigned instructed vocabulary conditions to classes and would have been coded as employing random assignment at the class level in reviews such as Plonksy and Gass’s (Reference Plonsky and Gass2011). Additionally, there have been divergent findings when it comes to the balance between random assignment at the class and participant levels. Plonsky (Reference Plonsky2013) observed that in both classroom and lab conditions, random assignment was more prevalent at the participant level. Farsani and Babaii (Reference Farsani and Babaii2020), however, observed that classroom experiments presented in Iranian theses had more group-level random assignments than person level with the opposite trend observed in lab conditions. Lindstromberg’s (Reference Lindstromberg2016) assessment of Language Teaching Research found that random assignment was overwhelmingly at the class level. Taken together, these syntheses paint a mixed picture with regard to random assignment at the participant and the class levels.

Despite the historical (Fisher, Reference Fisher1935) and current (Rogers & & Révész, Reference Rogers, Révész, McKinley and Rose2020) case for participant-level random assignment as the gold standard, some have argued that nonrandom procedures are also suitable. Farsani and Babaii (Reference Farsani and Babaii2020), for instance, coded studies that undertook purposeful assignment. When employing this strategy, as highlighted by Farsani and Babaii, L2 researchers consider tangential aspects such as L2 proficiency during the group construction process (e.g., ensuring groups are not significantly different in IELTS scores). Such procedures, however, violate the decades-old expectation that researchers do not introduce bias into the sample with their choices (Fisher, Reference Fisher1935). Furthermore, purposeful assignment becomes unnecessary when linear mixed-effects models (LMMs) and multivariate models with covariates are utilized. These analyses partial out constructs (i.e., covariates and random effects) that are tangential with the experimental condition(s) in predicting the outcome L2 IVA variable (Brysbaert & Stevens, Reference Brysbaert and Stevens2018).

Multisite Sampling

Recruiting participants from the intended population across two or more institutions or locations constitutes a multisite sample. The process of recruiting from multiple locations relates to the need for randomized sampling in quantitative research (Vitta & Al-Hoorie, Reference Vitta and Al-Hoorie2021). Moranski and Ziegler (Reference Moranski and Ziegler2021) argued that single-site samples present an external validity limitation, which reduces the ability to generalize findings beyond the context in which the data was collected. The authors also connected multisite samples with power, whereby the use of the former improves the latter. Such connections help reinforce the focus of this current study. Randomized samples from meaningful populations will additionally be multisite and thus the two constructs are related. For example, if a researcher intended to construct a sample that was representative of Saudi university EFL students and randomly selected 100 of such students from a national database, the probability of all participants’ being from the same university approaches zero.

A further benefit of multisite sampling is the facilitation of a “random(effect)-by site” analysis (Morgan-Short et al., Reference Short, Marsden, Heil, Issa, Leow, Mikhaylova, Mikołajczak, Moreno, Slabakova and Szudarski2018, p. 408). Under ideal conditions, the effect of the location should approach zero with experimental grouping variables and theory-driven covariates accounting for the dependent variable’s variance. Checking for this effect of location corresponds to the call for L2 researchers to consider clustering effects within their samples and data, whereby the different locations account for influence over the model estimates and parameters (Al-Hoorie & Vitta, Reference Al-Hoorie and Vitta2019). Although a multisite sample in L2 research would most likely be a nonprobability convenience sample conceptually, a random-by-site cluster check moves samples away from the biases that single sites introduce. As highlighted by Vitta and Al-Hoorie (Reference Vitta and Al-Hoorie2021), multisite sampling represents a compromise in relation to respecting the historical need for randomized sampling (Fisher, Reference Fisher1935) and recent L2 studies investing in the process (e.g., Hiver & Al-Hoorie, Reference Hiver and Al‐Hoorie2020).

Convenience samples from one location, regardless of the observed sample size, therefore have an inherent representativeness limitation in relation to generalizations beyond the setting from which they came (Moranski & Ziegler, Reference Moranski and Ziegler2021). While it is unreasonable to expect an L2 researcher to have access to national databases, multiple sites that are chosen with some randomness are perhaps within the realm of possibility. In the seminal reviews that started the L2 methodological reform (e.g., Plonsky, Reference Plonsky2013; Plonsky & Gass, Reference Plonsky and Gass2011), the need to consider power and thus construct robust samples was presented, but overt cataloguing of single- versus multisite samples appears to have been omitted until recently. Vitta and Al-Hoorie (Reference Vitta and Al-Hoorie2021) observed that 10 (18%) of 56 flipped learning experimental designs employed multi-site samples. Furthermore, in a review of L2 research practices, Farsani and Babaii (Reference Farsani and Babaii2020) noted that most of the observed samples were convenience samples, but the single- versus multisite issue was not assessed. From such limited data points, one cannot generalize more broadly about the state of multisite sample research in SLA. This view is somewhat reinforced by the observation that Moranski and Ziegler (Reference Moranski and Ziegler2021) and Morgan-Short et al. (Reference Short, Marsden, Heil, Issa, Leow, Mikhaylova, Mikołajczak, Moreno, Slabakova and Szudarski2018) presented position pieces with examples on the issue as opposed to dedicated methodological syntheses, a gap in the literature that the present study intends to fill by analyzing single- and multisite sampling in L2 IVA research.

Report Pool Construction

The report pool consisted of 110 studies retrieved from six renowned journals that were selected using a multistage systematic process. The first stage involved identifying candidates from a list of 35 field specific journals (Egbert, Reference Egbert2007) that were (a) SSCI indexed since 2010, (b) broad in focus (e.g., not ReCALL), and (c) targeted (in part) toward an audience of practitioners or agents of language teaching. SSCI indexed journals were considered important because Al-Hoorie and Vitta (Reference Al-Hoorie and Vitta2019) demonstrated that the quality of the statistical analyses is generally higher in such journals. This stage resulted in six candidate journals. The second stage involved a comparison of the six journals with two lists of SSCI journals from quantitative L2 research (Al-Hoorie &Vitta, Reference Al-Hoorie and Vitta2019; Plonsky & Gass, Reference Plonsky and Gass2011). Following this stage, one journal, ELT Journal, was removed for not being on both lists. In the final stage, three external experts confirmed that the five remaining journals could be labeled as “SSCI-indexed journals that are well-known, somewhat trusted in relation to their quantitative papers, and intended for practice-level research.” The five journals were Language Learning, Language Teaching Research, The Modern Language Journal, System, and TESOL Quarterly and they were employed to construct the report pool for Nicklin and Vitta (Reference Nicklin and Vitta2021). Subsequent expert review (two independent expert judgments) suggested that Studies in Second Language Acquisition should be included as its scope and prestige in the field mirrored that of Language Learning, which was included in the original report pool. To validate the inclusion, this rationale and the process of selecting the original five journals was presented to two other experts (the final external expert review) who concurred with adding Studies in Second Language Acquisition while also agreeing with the process and rationale for the selection of the five original journals.

From the six nominated journals, 1,053 reports, published between January 1, 2000 and December 31, 2020 were initially collected. The reports were retrieved from the EBSCO databases using the following search terms: vocab*, lexi*, idiom*, collocat*, phrasal, multiword, and formula, where * represented a “wild card” search term for which any character was valid. The EBSCO platform was employed for the search because individual journal platforms have inconsistent search options. EBSCO helped to systematize this process of the study (for a similar rationale for using Scopus to search journals; see Vitta & Al-Hoorie, Reference Vitta and Al-Hoorie2020). Because the search was conducted with EBSCO host, advance online publications were omitted from the report pool. For incorporation in the final pool, the reports were required to include (a) dependent variables measuring student acquisition of a set of target vocabulary items, (b) measurements enabling the dependent variables associated with L2 IVA (e.g., a vocabulary knowledge test), and (c) vocabulary treatment and comparison conditions that could be expected to occur in the classroom. When applying criteria (b) and (c), psycholinguistic measurements such as reaction times and very short treatments (e.g., 3,000 ms) were excluded as they were outside the intended L2 IVA focus. Because the report pool was originally constructed for another study, the reports were also required to contain sufficient information for effect size calculation. However, this final criterion had no bearing on the current study’s research questions. Analysis of the 1,053 reports, by the first author, resulted in a final pool of 110 that satisfied the three criteria.

The final report pool consisted of 110 reports from six journals: Language Learning (k = 11), Language Teaching Research (k = 39), The Modern Language Journal (k = 9), System (k = 18), Studies in Second Language Learning (k = 19), and TESOL Quarterly (k = 14). Supplementary Materials A contains a list of the studies comprising the report pool. A randomly selected group of 148 reports (from the 1,053) were independently coded by a second researcher to validate their inclusion in or exclusion from the final sample and confirm the criteria judgments. This process resulted in 89.80% agreement (κ = .80), which became 100% following deliberation.Footnote ⁴

Instrumentation and Coding

The present study’s first three research questions were operationalized as a series of dichotomous judgments regarding each report (see Table 1). These judgments comprised the study’s instrumentation. RQ1 involved judgments about the use of effect-size driven a priori sample-size planning procedures and procedures besides a priori ones. RQ2 necessitated judgments about (a) random assignment at either the participant or group level, and (b) randomized sampling. The judgment concerning the use of multisite samples in the operationalization of RQ3 required nuance. Some reports presented subsamples or sequenced experiments. In these cases, it was possible to conceptualize each report as a singular design. Accordingly, they were coded as employing multisite samples for respecting the bias that single sites introduce (Moranski & Ziegler, Reference Moranski and Ziegler2021). For example, Hulstijn and Laufer (Reference Hulstijn and Laufer2001) investigated reading, reading + fill-in, and writing interventions with Dutch and Hebrew L1 learners of English. This study was coded as multisite despite the inferential tests being confined to single settings. RQ3 also involved a dichotomous, complementary judgment regarding the consideration of cluster effects between multiple sites.

TABLE 1.

Coding Questions, Interrater Agreement, S index, and Notes

Note. RQ = Research Question

Using this framework, the primary researcher analyzed each of the 110 reports from the pool and dichotomously coded them as “1” for yes or “0” for no in relation to the questions listed in Table 1. Twenty-two (20%) randomly selected reports were independently coded by two researchers (the primary and a secondary researcher). The overwhelming frequency of no judgments prohibited inferential assessments using Cohen’s κ. As highlighted by McHugh (Reference McHugh2012), κ statistics are driven by the chi-squared test and an abundance of no judgments unbalance the contingency table rendering the test inappropriate. In response to this limitation of κ, the S Index was employed to assess observed interrater reliability. Norouzian (Reference Norouzian2021) presented the SLA research community with a turnkey program to calculate the S Index metric for interrater reliability (Falotico & Quatto, 2010, 2015, as cited in Norouzian, Reference Norouzian2021), which like κ partials out change agreement but is not sensitive to the imbalance that renders κ unusable in cases found in this current study (see Figure 3, Norouzian, Reference Norouzian2021, p. 6). As highlighted in Table 1, all judgments had high (.80 to .89) to very strong (.90 to 1.00; ‘NA’ denotes perfect agreement with only one or zero ‘yes’ judgments) observed S Index values according to Norouzian’s stated magnitude thresholds, except for random assignment at the participant level (RQ 2b; S Index = .73; “acceptable”). The relatively low value for the random assignment at the participant level judgment could be attributed to the ability to engage in the design process without having to overtly label it as “random assignment.” Some report authors engaged in the design feature without overtly employing the “random” label (e.g., Traxler & Nakatsukasa, Reference Traxler and Nakatsukasa2020) while others noted random assignment in past studies but omitted detailing the feature in their methodology (e.g., Choi, Reference Choi2017). Power considerations, conversely, do not facilitate such variation in wording when presented in the methods.

In line with past L2 meta-analytic reports (e.g., Al-Hoorie & Vitta, Reference Al-Hoorie and Vitta2019; Nicklin & Vitta, Reference Nicklin and Vitta2021; Vitta & Al-Hoorie Reference Vitta and Al-Hoorie2020, Reference Vitta and Al-Hoorie2021), the primary and secondary researchers who coded the 22 reports for the reliability check discussed disagreements to identify possible systemic issues in the coding process and to identify points of clarification. Such discussion resulted in agreement with the primary researcher’s judgments and rationale behind them being accepted by the second coder while highlighting points of clarification such as double site studies where inferential tests were constrained to single sites being coded as multisite as highlighted in the preceding text.

Analysis

In addressing the first three research questions, each of the 110 reports from the pool was coded according to the seven questions in Table 1, and the tallied responses (judgments) are presented visually. To address RQ4, which related to changes in sample-size planning, randomization, and multisite sampling over a 20-year period, the results for each of the dichotomous judgments (or aggregates of them) were correlated with the year of publication. To frame the correlation values further, an analysis of report publication by year was also conducted. Simple regressions with year as the dependent variable and a series of dummy variables representing each coded binary judgment as the predictors were conducted. Visual inspection of scatterplots, with standardized residuals on the y-axis and standardized predicted values on the x-axis, suggested that the homoscedasticity assumption might not have been met because cases unevenly straddled the x-axis (see Osborne & Waters [Reference Osborne and Waters2002] for using residual scatterplot to check this assumption). This was unsurprising given the extreme frequency of “no” judgments, thus Spearman’s rho (ρ) tests were conducted. Dellinger (Reference Dellinger and Allen2017) summarized the assumptions of Spearman’s ρ as (a) paired observations, (b) ratio, interval and/or ordinal variables, and (c) a monotonic relationship (i.e., no change in direction), which are less stringent assumptions than those associated with parametric correlation testing (see Field, Reference Field2018). In presenting the case to use ρ with binary (1/0) and ranked variables, Glass (Reference Glass1965) highlighted the procedure’s “weak” assumptions in comparison to parametric assumptions as an advantage for extreme distributions of variables (e.g., the overwhelming frequency of “no” judgments across reports in this current study). Spearman’s ρ and accompanying 95% confidence intervals (CIs) were calculated using the spearmanCI package (de Carvalho, Reference de Carvalho2018) in R (R Core Team, 2020).

Finally, the possibility of a by-journal clustering effect was assessed with Fisher’s (for 2×2 contingency tables) and Fisher-Freeman-Halton (for contingency tables exceeding 2×2) Exact testing, despite the conceptual justification for viewing them as being a homogenous group (see preceding text). A cluster effect within the journals occurs if reports from one or more of the six journals comprises a subset that is more likely than the rest of the journals to include one of the three aspects of research design under investigation. Because the expected cell count was less than five in more than 20% of the cells, Χ² testing was not employed, as recommended by Field (Reference Field2018). Nonsignificant associations between the journals and the dichotomous judgments operationalizing the first three research questions suggested that clustering concerns were mitigated.