“A call for cautious interpretation of meta-analytic reviews” by Frank Boers, Lara Bryfonski, Farahnaz Faez, and Todd McKay

We would like to express our gratitude to the review board of SSLA for selecting our article (“A call for cautious interpretation of meta-analytic reviews,” March 2021) for this year’s Albert Valdman award for outstanding publication. We did not expect this publication would attract so much attention. In the article, we demonstrate that the bottom line of a meta-analysis is determined by the methodological choices made by the meta-analyst, including how effect sizes are calculated, how factors are defined, and how the eligibility of primary studies is determined for inclusion in the analysis. We illustrate this by revisiting three recent meta-analytic reviews (about pragmatics instruction, the use of corpora for vocabulary learning, and the effectiveness of task-based language programs) and by closely inspecting various aspects of their methodologies. After reanalyzing the data, or applying more stringent inclusion criteria, some of the original conclusions are no longer supported. We assume that our article attracted special attention because the bottom lines of our three replications differ so markedly from the original ones. Our replications point out the factors researchers should consider when deciding whether a pertinent area of pedagogy-oriented applied linguistics is ripe for informative meta-analysis.