Objectives
The primary aim of this seminar was to promote the values and practices of research synthesis (a systematic form of secondary research – see Chong and Plonsky (Reference Chong and Plonsky2021a) for a typology of different forms of research synthesis) in applied linguistics and language education in advancing research and research-practice conversations. Moreover, as a seminar that had a methodological rather than substantive focus, another objective was to discuss methodological traditions and issues of conducting different types of research synthesis, including meta-analysis, qualitative research synthesis. Being cognizant of the fact that research synthesis methodologies in applied linguistics are still in their infancy and draw on eclectic methodological traditions in other disciplines (e.g., healthcare and medicine), an added value of this seminar was to facilitate interdisciplinary dialogues by inviting research synthesists in other disciplines to share their perspectives and experiences. This seminar attracted a total of 97 participants.
1. Day 1: Research synthesis in applied linguistics
In his keynote ‘Measuring up: Psychometric concerns and contributions of meta-analysis’, Dr. Luke Plonsky (Northern Arizona University) shared four ‘moments’ that underscore the intersections between meta-analysis and measurement, including outcome measure as moderator of meta-analytic effect, validity assessment, correction of meta-analytic effects, and rater reliability.
Dr. Graeme Porte's (University of Granada) invited presentation focuses on publishing literature reviews and survey studies in Cambridge University Press's international journal, Language Teaching. Offering his advice as the journal editor, Dr. Porte introduced the various sections of the journal and invited submissions that report synthetic findings relevant to applied linguistics and language education in various forms. Following Dr. Porte's presentation was a session delivered by Dr. Vahid Aryadoust and Azrifah Zakaria (National Institute Education, Nanyang Technological University, Singapore) where they presented a methodological framework of conducting scientometric review, an emergent type of research synthesis in applied linguistics.
The afternoon began with a presentation by Ekaterina Sudina (Northern Arizona University) on the choices and challenges of coding for validity evidence of second language (L2) research instruments. Sudina provided suggestions for assessing validity evidence of scales used in primary studies, which were exemplified in a comprehensive coding scheme comprising 82 features. Fiona Victory's (University College London) presentation on her comparative analysis on medium of instruction (MOI) choices introduced the qualitative facet of research synthesis. In her presentation, Victory provided an overview of a synthetic method called ‘qualitative comparative analysis’ and shared how she applied this method to analysing documentary data to identify cross-national patterns and trends in MOI choices in schools.
The first day concluded with the announcement of a plan to set up a new British Association of Applied Linguistics (BAAL) special interest group (SIG) on research synthesis and secondary research. I met with potential members to discuss rationale, goals, and plans of the SIG, and a proposal draft was shared with the interested participants in the presentation to elicit feedback. It was suggested that the SIG could develop methodological guidelines and share best methodological practices of conducting various types of research syntheses in applied linguistics. Among its other aims, it is hoped that this SIG can promote research synthesis as a diverse repertoire of research methods comprising not only quantitative (i.e., meta-analysis e.g., Lee et al. Reference Chong and Reinders2015) but also qualitative techniques (i.e., qualitative research synthesis e.g., Chong & Plonsky, Reference Chong and Plonsky2021b).
2. Day 2: Interdisciplinary practices of research synthesis
The second day of the seminar included presentations by synthesists in applied linguistics and other research disciplines. Embracing this interdisciplinary spirit, Dr. Talia Isaacs (University College London) presented her keynote entitled ‘Using systematic review methodology in health intervention research: Applications for applied linguistics’. Research notions in healthcare such as priority-setting, patient and public involvement were introduced and similarities with foci of applied linguistics research were drawn (e.g., conducting practitioner or classroom-based research). Drawing on works by Dr. Iain Chalmers, Dr. Isaacs argued that systematic reviews and meta-analyses are useful to reduce ‘research waste’ and address the above notions.
In his presentation, Dr. Hamish Chalmers (University of Oxford) introduced an initiative he led: an online platform called The International Database of Education Systematic Reviews (IDESR) where synthesists in L2 education can register their research protocols. Dr. Chalmers shared ways researchers can register their protocols on IDESR and how they can benefit from the platform. Dr. Rita Silver (National Institute Education, Nanyang Technological University, Singapore) and her Singapore-based team delivered an inspiring talk on conducting various forms of synthetic reviews for practitioners and policymakers. In particular, Dr. Silver shared the strategies they employed to disseminate research findings to stakeholders who are not researchers in an accessible manner.
Focusing on systematic reviews in educational research, Professor Sarah Miller, Director of Campbell UK & Ireland gave a presentation on supervising masters and doctoral students who conduct systematic reviews as (part of) their dissertations. Professor Miller suggested ways that supervisors can guide their students to pay attention to core characteristics of desk-based research, especially those related to methodological soundness. Based in the United States, Dr. Taichi Yamashita (Iowa State University) shared his work in progress, a systematic review on arguments in favour of replication research found in the journal, Language Teaching. Dr. Taichi identified modifications suggested by researchers and discussed major motivations for researchers recommending replication.
Also focusing on postgraduate dissertations, Dr. Mohammad Amini Farsani (Iran University of Science and Technology) presented a methodological review on MA theses conducted by students in Iran in the past three decades. Focusing on quantitative and mixed-methods theses, methodological strengths and weaknesses were discussed. The final presentation by Andrea Vaughan (University College London) demonstrated versatility of research synthesis methodologies. Vaughan introduced her research applying systematic review methods to synthesising corpus linguistic research. A specific focus of her presentation was the use of an appraisal tool to analyse corpus linguistic studies.
To conclude the two-day seminar, speakers and audience engaged in vibrant and fruitful discussions related to issues and methodologies of research synthesis. Some topics discussed were: the possibility of establishing a journal on research synthesis in applied linguistics, and ways to incorporate research synthesis into postgraduate research methodologies modules.
3. Conclusions
The aims of the seminar are encapsulated in its name ‘Research Synthesis in Applied Linguistics: Facilitating Research-Pedagogy Dialogue’, which include a dual focus on methodology and application. Methodologically, the seminar demonstrates to the applied linguistics community the need to develop clearer, discipline-/field-specific methodological guidelines (see an example by Chong & Reinders, Reference Chong and Reinders2021). With regards to application, some presentations addressed an under-explored area of research synthesis that has to do with research dissemination and accessibility. These presentations emphasised the fact that research synthesists are responsible for conducting reviews which not only are methodologically rigorous but also can be easily accessible by researchers and other stakeholders (e.g., practitioners) (Chong, Reference Chong2019).
The takeaway of this seminar can be summarised as follows:
• Research synthesis is not (only) meta-analysis; it is a repertoire of quantitative, qualitative, and mixed research methods.
• Research synthesis is not traditional literature review; it emphasises systematicity and replicability in methodology.
• Research synthesis is not only for researchers; it is potentially useful to teachers and policymakers.
• Research synthesis needs to be considered not only as discipline-specific research methods but also for its wider applications in other disciplines.
Hopefully, this seminar has begun a conversation in this emergent research methodology in applied linguistics. The next step is to develop platforms to continue such professional dialogue and to promote the use of the methodology in various subfields in applied linguistics. For instance, meta-analyses dominate L2 acquisition studies (e.g., Teimouri et al., Reference Lee, Jang and Plonsky2019), while qualitative research syntheses are more commonly found in the field of computer-assisted language learning (e.g., Chong & Reinders, Reference Chong and Reinders2020). Some initiatives that stemmed from the seminar are already underway, including the plan to set up a new BAAL SIG on research synthesis in applied linguistics and a special issue on research synthesis to be published in the journal Innovation in Language Learning and Teaching (Taylor & Francis).
