1. Introduction
This paper presents and discusses three empirical studies in the domain of second or foreign language (L2) listening comprehension that deserve to be replicated. The under-researched nature of this domain did not offer a wide range of studies to consider. However, given the importance of listening comprehension in second language acquisition (SLA), replicating research, particularly studies on successful practices in listening pedagogy, as well as multimedia applications, would be warranted. Further work in this area is particularly salient because of the strong potential for improvements in L2 listening pedagogy (to facilitate L2 acquisition) as well as research methodology in this domain. Of the three papers we have selected for discussion, one focuses on investigating strategy instruction – Graham & Macaro (Reference Graham and Macaro2008), and another on metacognitive instruction – Vandergrift & Tafaghodtari (Reference Vandergrift and Tafaghodtari2010). Both of these relatively recent studies were published in Language Learning and fill important gaps in their respective areas of listening comprehension research. The third paper, by Brett (Reference Brett1997), an early study published in System, has promoted initial interest in, and understanding of, researching multimedia applications for listening comprehension. The studies represent important contributions to the field and merit replication to test the robustness of the original study.
In sections 2–4 of this paper, we firstly provide a brief background to the aspect of L2 listening comprehension research covered. Then, for each of the three studies, we overview the original study and propose reasons for and approaches to replication, both approximate and conceptual. Lastly, we present our concluding thoughts.
2. Listening: Strategy instruction
Listening strategy instruction is a strand of L2 listening pedagogy research which has a comparatively long history of over 25 years. It is premised on the notion that less-skilled language learners can be taught to emulate the productive language learning behaviours of their more-skilled peers. Learners are trained through explicit techniques such as raising awareness of strategies already used; presenting and modelling strategies; providing multiple opportunities for practice with gradual withdrawal of teacher scaffolding; and evaluation of effective transfer of strategy use to new tasks (Cohen Reference Cohen and Hinkel2011).
Few training intervention studies have been published, particularly for listening. The research design employed in such studies has largely been quasi-experimental and has focused on the explicit teaching of one or more strategies in isolation or along with strategies for other language skills (see, for example, O’Malley & Chamot Reference O’Malley and Chamot1990). These studies tend to report positive results for the use of only one or two strategies, and only immediately following the instruction period. There is no clear and established listening strategy pedagogy, and the research evidence for the long-term benefits of strategy instruction remains inconclusive. Furthermore, listening strategy instruction research has been criticised for not controlling for potential confounds such as language proficiency level, and being unable to identify a causal relationship between instruction and comprehension gains (Macaro, Graham & Vanderplank Reference Macaro, Graham, Vanderplank, Macaro and Cohen2007).
Listening strategy instruction has not delivered consistently successful outcomes and a number of research-related issues remain. Therefore, before we move further ahead with additional research in this area, there is a need to replicate key studies that have delivered positive results. In particular, we need to replicate studies that demonstrate long-term results in improving listening proficiency (see Plonsky Reference Plonsky2011), to test the robustness of these results, verify the nature of the intervention and more closely examine the context and the circumstances that may have led to these results. This kind of information will lay more solid foundations for future listening pedagogy and offer greater confidence in the robustness and the generalisability of the results. The success of the intervention reported by Graham & Macaro (Reference Graham and Macaro2008), the nature of the study (short- and long-term results), the link with self-efficacy, its rigour (as suggested by the quality of the journal in which it is published), and that it has been cited in over 110 subsequent related publications, make this study worthy of replication to test the robustness of the results.
2.1 Suggested study for replication: Graham & Macaro (Reference Graham and Macaro2008)
2.1.1 Background to the study
This study explored the effects of listening strategy instruction with secondary-level learners of L2 French in England, as well as the effects of the intervention on the participants’ self-efficacy (beliefs about their own abilities, perceived success or lack of it) for listening comprehension. Based on a pre-intervention study, Graham & Macaro hypothesised that learners would benefit most from an intervention focusing on a cluster of strategies that would help participants to form effective predictions, confirm evidence for these predictions, identify key words, infer the meaning of unknown words, recognise familiar words and recognise word boundaries. The intervention took place over a period of six months and also involved a delayed post-test six months later, another important contribution of this study.
The study design was quasi-experimental with a pre-test, post-test, and a delayed post-test with two experimental groups and a control group. The sampling strategy involved a stratified sample of matched pairs for a total of 151 learners from 15 similar schools divided into three groups. Complete data sets were available for 107 participants for the pre-/post-phase and for 59 at the delayed post-phase. The comprehension test involved listening to three different audio texts twice on the same topic, after which participants wrote free-recall protocols based on their understanding of each text. The two experimental groups were divided into a high-scaffolding group and a low-scaffolding group. These groups received relatively explicit strategy training that involved consultation of strategy lists; awareness-raising materials for bottom-up processing and word segmentation strategies for French; materials for accurate inferencing; prediction and evaluation strategies; and self-evaluation materials. The low-scaffolding group was limited to these activities, whereas the high-scaffolding group engaged in further reflection through the completion of a diary and written feedback on strategy use. The researchers provide more detail on the different aspects of the intervention; however, it is not clear whether teachers received training so that the intervention was consistently delivered to the different classes that made up the two experimental groups. The intervention was monitored selectively by the researchers, though it is not clear whether the control group was monitored in any way.
Data analysis revealed that the intervention was successful. First, listening test responses were scored independently by two raters to determine the number of idea units recalled for each audio-text. Inter-rater reliability for the pre-test and post-test was 0.95 and 0.96 respectively. The reliability co-efficient for the delayed post-test scores was not provided. An ANCOVA was carried out to determine any significant differences between the pre-test and post-test scores, in order to control for any initial differences between the control group and the experimental groups. The intervention proved to be significantly different (p < 0.001) with a small effect size (0.19). Results for the delayed post-test were similar, demonstrating the long-term effects of the intervention. Differences between the high-scaffolding and low-scaffolding groups after the intervention revealed a statistically significant advantage for the high-scaffolding group (effect size = 0.25). Long-term effects of the intervention showed that the low-scaffolding group outperformed the other two groups; however, no effect sizes were given. The intervention also had salutary effects on student self-efficacy beliefs for listening comprehension, as measured at the completion of the intervention.
2.1.2 Approach to replication
This study deserves replication because of its theoretical motivation (explicit teaching of a repertoire of listening strategies) and the apparent success of the intervention. However, given that the methodology cannot be replicated exactly, we would suggest a conceptual rather than an approximate replication. A conceptual replication would allow for an experimental intervention that is built on the same theoretical underpinnings underlying the study, allow for some adjustments in methodology and yet provide greater insights into the results of the original study.
First of all, the listening tests could be standardised for all three testing sessions. The researchers cite the nature of the testing procedure (free recall written protocols) as an important methodological feature of the study, arguing that it did not duplicate a listening test type practised during the intervention and, therefore, demonstrated the ability of the learners to transfer the strategies to new task types. However, although the researchers argue that measures were taken to ensure a close match between each test, the reliability was not verified and we cannot be sure that each test was comparable, in spite of the ‘in-house’ attempts to hold constant the level of difficulty and to ensure a high level of inter-rater reliability between test scorers. On the other hand, using a reliable and valid listening test for L2 French learners, constant from one session to the next, would ensure more reliable scores for each testing session. We would argue that a standardised listening test with a variety of question type responses (e.g., previous versions of the DELF, Diplôme des études en langue française) would still demonstrate the learners’ ability to transfer strategies to a new situation and provide more reliable data for determining the success of the pedagogical intervention. This standardised test, with demonstrated reliability, would not be limited to the three topics chosen for the original study.
Secondly, a conceptual replication would allow us to control for potentially confounding variables. In order to control for the teacher variable, we suggest that a control group and an experimental group be established in each school with the same teacher teaching both groups. Each teacher should be observed regularly to ensure distinct delivery of the pedagogy targeted for each group. The researchers do note that the minimal observation of the experimental intervention in the range of classes in the original study does not allow them to claim categorically that the intervention was delivered uniformly. However, they argue that this limitation could be considered a strength if researchers could demonstrate the success of the experimental intervention in diverse academic environments. However, if we control for the particularly powerful variable of teacher, the results of the original study would likely be more reliable and could be interpreted with greater confidence.
Finally, a conceptual replication can be further enriched by adding a qualitative component. Use of a questionnaire to tap learner awareness of the processes underlying listening comprehension (e.g., Vandergrift et al. Reference Vandergrift, Goh, Mareschal and Tafaghodtari2006) could provide insight into the changes underlying the success of the intervention. Furthermore, a think-aloud methodology could be used with a representative sample of the experimental group to examine any significant differences (e.g., two or more points) in item responses pre- and post-intervention. The insights provided by qualitative methods would tell us more about the original study by unveiling listening behaviours that were most affected by the intervention and offer further insight (student ‘voice’) to these changes as students attempted to resolve difficulties encountered in real-life listening tasks.
3. Listening: Metacognitive instruction
Metacognitive instruction is a relatively recent research strand in L2 listening pedagogy. It is underpinned by the extensive literature on metacognition in learning (e.g., Hacker, Dunlosky & Graesser Reference Hacker, Dunlosky and Graesser2009) and, with regard to L2 listening in particular, by the theoretical principles presented by Vandergrift & Goh (Reference Vandergrift and Goh2012). These authors outline a pedagogical approach that is more holistic than strategy instruction, focusing on a range of techniques to help learners increase their awareness of listening comprehension processes and their ability to plan, monitor and evaluate their listening learning.
In terms of related empirical research, not many studies have been conducted, mostly because interest in the role of metacognition in L2 learning is relatively incipient. Nonetheless, the few published studies to date have found that metacognitive instruction can have a positive impact on learners’ listening comprehension ability, particularly for less-skilled listeners (e.g., Cross Reference Cross2011). These studies typically employed a pedagogical cycle similar to that advocated by Vandergrift (Reference Vandergrift2003), which is believed to implicitly develop learners’ comprehension when task-embedded strategies are cued by the sequence of task execution. However, it is still not clear how or why this pedagogical cycle actually brings about performance gains. In addition, there has as yet been no attempt to establish which configuration of possible techniques proposed by Vandergrift & Goh (Reference Vandergrift and Goh2012) can optimise listening development. The study by Vandergrift & Tafaghodtari (Reference Vandergrift and Tafaghodtari2010) enlarges the theoretical framework of strategy instruction by offering a sound and systematic early attempt at investigating the effects of metacognitive instruction. With already more than 132 citations in other related publications, this study would be worthy of replication to verify the robustness and generalisability of the results so that we can build on this study and better understand how best to teach L2 listening comprehension.
3.1 Suggested study for replication: Vandergrift & Tafaghodtari (Reference Vandergrift and Tafaghodtari2010)
3.1.1 Background to the study
This study investigated the effects of a pedagogical approach to facilitate successful L2 listening comprehension; in this case, a metacognitive ‘process-based’ approach to teaching L2 listening that involved more implicit instruction in listening processes. Based on earlier studies (e.g., Vandergrift Reference Vandergrift2003) demonstrating the short-term effects of this approach on learner motivation and the development of metacognitive knowledge about listening, the researchers hypothesised that the experimental group receiving metacognitive instruction would outperform the control group and that the less-skilled listeners in this group would make greater gains in listening achievement than their more-skilled counterparts. Both groups also completed the Metacognitive Awareness Listening Questionnaire (MALQ, Vandergrift et al. Reference Vandergrift, Goh, Mareschal and Tafaghodtari2006); however, our argument for replication will focus primarily on the pedagogical intervention. The study took place over a 13-week semester with university students learning French. The researchers attempted to carefully control for the variables that could impact the validity and reliability of the research.
The study design was quasi-experimental with a pre-/post-test design. In this case, there was no delayed post-test. Participants came from six intact classes, randomly assigned by class to either a control or an experimental group. Participants were identified as less- or more-skilled listeners on the basis of their score on the listening pre-test, also used as a post-test to measure growth in listening ability. Two instructors participated; one taught four different classes of low-intermediate learners and the other taught two different classes of high-beginners. To control for potentially confounding teaching variables, the same teacher taught both the control and the experimental group, and both groups listened to the same texts. All teaching sessions were observed by a research assistant to verify that the instructor respected the designated listening pedagogy for that group. Each week, the classes listened to a different oral text related to the topic of the teaching unit. Participants in the control group listened to the same texts the same number of times; however, they did not engage in prediction; they were not given an opportunity to discuss, predict or monitor their comprehension with a classmate; and they did not engage in any whole class reflection on strategy use. The steps in the pedagogical intervention experienced by the experimental group, and the metacognitive processes underlying each step in the cycle, are described in detail in the paper.
Data analysis revealed that the intervention was successful. In order to confirm the hypotheses, a 2-factor ANCOVA was performed. Independent variables consisted of group (treatment and control) and the level of listening ability (less- and more-skilled), factorially combined. To control for any initial differences in listening ability, pre-listening test scores were used as covariate in the analysis. Results revealed that the experimental group outperformed the control group (p < 0.05), with a fairly strong effect size (0.30). The less-skilled listeners made greater gains than their more-skilled peers (p < 0.05); however, the effect size was minimal (0.01).
3.1.2 Approach to replication
We suggest that this study be replicated because the experimental listening pedagogy appears to be successful in improving listening achievement. Given that the intervention is clearly described, an approximate replication would be relatively easy to carry out and would verify the strength of the results. Furthermore, replicating this study with different learner groups and in different learning contexts (e.g., language, age, proficiency level, learning context) will allow us to verify the generalisability of the results of the original study to new populations and settings.
In addition, we would suggest a number of possibilities for a conceptual replication of this study. These options would respect the intervention and theoretical underpinnings of the original study, verify those results, and yet add another layer of interest. First, the original study was very ‘top-down’ in nature, in that it emphasised the use of predictive strategies to develop a conceptual framework for interpreting the oral text. As proposed by the authors of the original study (p. 491), we suggest adding a ‘bottom-up’ dimension where learners would be given a transcription of all or part of the text that they would follow during the third listen to a playing of the text. This would help learners make sound-form connections with the portions of the text that they found difficult to decipher. In this case, we suggest two experimental groups (one of which would replicate the experimental group of the original study) along with the control group. Second, a pre-/post-listening questionnaire (e.g., MALQ) could be introduced to examine the overall change in reported strategy use for insight into the success of the intervention. Third, the self-efficacy dimension of the Graham & Macaro study would also add greater interest and importance to the replication. It would be useful to ascertain whether the pedagogy of the original study helps learners to better regulate their comprehension processes and perceive listening success as something within their control. Building on Graham (Reference Graham2006), self-efficacy beliefs should improve; that is, learners’ beliefs regarding their own ability to be successful listeners should grow and, concomitantly, their motivation to engage in future listening tasks. Finally, a delayed post-test, context permitting, would also add an important element to the replication in order to determine the long-term effect of the intervention. In order to retain the focus on replicating, and comparing results with, the original study, however, only one of the two above-mentioned suggestions should be added to the conceptual replication at a time. By minimising the addition of new methodological features, focus on the original study is optimised. Any new data, however, can potentially shed light on either the reasons for the success of the intervention or supplementary effects of the intervention.
4. Listening: Multimedia applications
The move towards the greater use of multimedia applications for L2 listening comprehension has mirrored the significant increase in utilising Computer Assisted Language Learning (CALL) in classrooms over the past 15 years. This transition has promulgated ongoing interest among L2 listening researchers (e.g., Grgurović & Hegelheimer Reference Grgurović and Hegelheimer2007; Cárdenas-Clarosa & Gruba Reference Cárdenas-Clarosa and Gruba2012) in investigating appropriate and effective instructional design for taking advantage of the functionality (e.g., use of captions) and interactivity (e.g., learner control over help options) offered by multimedia applications. Thus far, the findings from such research have been mixed; improved listening scores were evident in some studies but not in others, with learners’ language proficiency and idiosyncratic use of technical attributes being influential factors.
These studies essentially focused on investigating learner choice of various functional and interactive features to aid their comprehension of a given text, with learners not given any training in how to best exploit the technology. Also, most studies did not address the impact of the functionality and interactivity of multimedia applications on learner development of listening comprehension skills. Moreover, despite enthusiasm for exploiting multimedia applications in L2 listening comprehension among researchers (and teachers), few studies have been conducted to establish whether multimedia applications with their attendant attributes actually provide a superior pedagogical platform (or not) compared to video-only or audio-only modes traditionally employed for developing listening comprehension ability. Nor has a concrete pedagogy for using multimedia applications in L2 listening been forthcoming.
Given the general concerns raised regarding previous research into the functionality and interactivity of multimedia applications, and that their capacity as a medium for teaching learners how to listen remains uncertain, it is important to understand more about whether, and in what way, multimedia applications can optimally contribute to supporting and developing listening comprehension. Therefore, the study by Brett (Reference Brett1997) has been selected for replication. It is an oft cited work in the field and one of the first to explore the issues that currently remain, particularly in terms of examining listening text delivery mode.
4.1 Suggested study for replication: Brett (Reference Brett1997)
4.1.1 Background to the study
This study investigated the relative effect on listening comprehension of computer-based multimedia by comparing English as a second language (ESL) users’ understanding and language recall of content presented through three separate media – audio, video and multimedia. At the time of the study, the use of CD-ROMs containing material for listening in another language via computers was beginning to emerge in suitably equipped classrooms and self-access centres. This study was correspondingly motivated by the then lack of associated empirical research regarding the utility of such multimedia applications. Brett aimed to ascertain whether learners’ comprehension levels and subsequent recall of specific language were better for multimedia than for audio or video. He also set out to explain the particular features of multimedia which may facilitate improved performance for the given tasks and to identify how multimedia was perceived as a listening and a learning mode.
Brett prepared six ‘transactional’ listening texts with three versions of each -- audio, video and multimedia (a total of 18 texts). The audio content was identical for the three versions of each of the six separate texts, and the visual material in the video and multimedia was also the same for a particular text. Three sets of quantitative data were then collected from 43 undergraduate, advanced-level users of English from various European countries in existing class groups. The first data set was the listening comprehension scores. Participants listened only once to two different texts presented through each of the three media (six discrete texts in total) and completed a listening comprehension task (true/false for five of the texts and sentence ordering for the one other text) as they listened. The second data set was the scores from a cloze test administered after the comprehension task. Participants completed cloze items in transcripts of the listening texts based on what they could recall from the previous listening. The final data set was the questionnaire responses regarding reactions to the features, efficiency and effect of listening using multimedia. This data set was gathered from a later follow-up session where participants had the opportunity to explore the multimedia material more comprehensively than during the initial listening task. A t-test calculation was used to identify any significant difference (p < 0.05) between multimedia listening comprehension and recall scores and those for audio and video for each of the six texts with three versions. Regarding comprehension scores, multimedia listening comprehension scores were at or approaching significance in 7 of 12 t-test calculations. As for the cloze test scores, the t-tests indicated that language recall was significantly better for multimedia in 5 out of 12 calculations. Questionnaire responses essentially indicated that functionality aspects of multimedia (e.g., question presentation and rapid feedback) were positively received in terms of facilitating comprehension.
4.1.2 Approach to replication
This study can be usefully replicated in several ways. Approximate replication is proposed to address the limitations of the original study, particularly those mentioned by Brett (p. 48). Initially, we suggest a ‘close’ approximation that attempts to mirror, as far as possible, the content and procedure of the original study but modifies several particular facets. Increasing the number of participants (who would ideally be representative of the undergraduates in the original study) and balancing their distribution across each delivery method is suggested. This would improve the reliability of the t-test calculations employed. In addition, increasing the number of true/false, sentence ordering comprehension, and recall cloze items would permit a larger data sample of performance and, thereby, more fully capture and represent participant ability. This replication with minimal variance would be a useful step towards establishing the veracity of the findings in the original study.
Other replications could also prove informative. In such replications, the procedural approach in the original study could be essentially adhered to, along with the modifications mentioned above. However, one crucial variation to the procedure that would need to be made for ecological validity is that participants should have the opportunity to attend to and process the simultaneous visual and audio content in the video and multimedia condition without distraction. As such, the completion of tasks should occur when the text is paused (not while listening) for all three media in any replication, with response feedback in the multimedia mode following as per the original study. Various facets, many suggested by Brett, could be usefully investigated in various combinations in broader replications to compare the effects of the three media. Examining participants of various levels of language proficiency, nationalities and backgrounds would make it possible to ascertain if the original study's findings are generalisable across different populations. Using different item types (e.g., short answer or written recall) than those in the original study for measuring comprehension and recall would help establish if varying this facet influenced the relative utility of multimedia previously illustrated. In addition, responses to item types such as written recall can better reflect participants’ actual understanding than, for example, true/false items which have a 50% chance of being correctly guessed. Likewise, with respect to functionality and interactivity attributes, investigating diverse forms of help options (e.g., brief audio or on-screen explanations, providing the transcript) would offer additional insights into whether this was a key factor in participant success rates for multimedia in the original study as claimed by the author, and add to the current knowledge base resulting from associated research since Brett's study. Exploring alternative listening text types (e.g., news broadcasts, dialogues) than that used in the original study would similarly further our understanding of how this facet may affect the comparative effectiveness of multimedia applications.
5. Conclusion
It is axiomatic to say that the role of listening in SLA needs greater research attention. The covert nature of the cognitive processes and ephemeral nature of the input make listening research more difficult and less attractive than research into the other language skills. This article, however, has discussed some fertile avenues for future replication research in L2 listening, reflecting on three studies that represent fields of L2 listening inquiry which have the potential for a powerful impact on listening pedagogy and, ultimately, successful language learning. Furthermore, the discussions on research methodology will enhance future L2 listening research design to ensure greater validity and reliability.
Larry Vandergrift is Professor Emeritus from the Official Languages and Bilingualism Institute of the University of Ottawa, Canada. He has published widely on L2 listening comprehension, including a recent book co-authored with Christine Goh (Routledge, 2012).
Jeremy Cross is an Associate Professor in the Graduate School of Languages and Cultures at Nagoya University, Japan. His primary research interest is L2 listening comprehension, and he has published related articles in various international journals.
