INTRODUCTION
The studies in this special issue, Language Learning from Multimodal Input, investigate the potential of multimodal input for language learning from different angles. They focus on different types of linguistic outcomes, such as grammar, vocabulary (both single words and formulaic sequences), pronunciation, and comprehension but also on L2 learners’ processing of multimodal input. In doing so, these studies examine research questions that have, so far, received little attention in the research literature and, thus, provide an important contribution to the field.
My commentary on this special issue consists of three parts. First, I discuss aspects related to the multimodality of input and the potential role of the multimedia learning theory (Mayer, Reference Mayer2001) in studies on multimodal L2 input. Next, I put forward three main reasons that may explain the importance of multimodal input for SLA researchers. I conclude this commentary by discussing the contribution of this special issue to theory, methodology, and pedagogy.
DEFINING MULTIMODAL INPUT
As language users, we are constantly surrounded by multimodal input (see Kress & Van Leeuwen, Reference Kress and Van Leeuwen2001 on multimodality). For instance, when we watch TV, we deal with incoming sound and images. In everyday conversations, we process sound, gestures, body language, and so forth. Being able to extract meaning from multimodal input is thus a crucial component of everyday life. Drawing on the multimodality of everyday life, the Douglas Fir Group (2016) has defined language learning as semiotic learning, which means that we can rely on a wide range of semiotic resources in our L2 meaning-making and learning processes, such as nonverbal information (e.g., facial expressions, body language), but also textual, graphic, or auditory resources. Language learning is thus necessarily multimodal learning.
During the last decade, we have witnessed a remarkable number of studies on the role of multimodal input for aspects of L2 acquisition. Even though studies already explored the potential of multimodal input in the nineties and the early 2000s (see Vanderplank, Reference Vanderplank2010 for a comprehensive review), we can clearly notice a renewed and reinforced interest into the role of multimodal input around 2009–2010. First of all, there is the publication of a number of corpus-based studies (e.g., Webb & Rodgers, Reference Webb and Rodgers2009; Webb, Reference Webb2010a, Reference Webb2010b) that highlight the potential of TV as an important source of L2 input by investigating the vocabulary demands of television programs such as TV series and films to determine target vocabulary sizes needed for adequate comprehension of those programs. Second, a number of empirical studies such as the ones carried out by Winke et al. (Reference Winke, Gass and Sydorenko2010) and Sydorenko (Reference Sydorenko2010) on captioned video or Brown et al. (Reference Brown, Waring and Donkaewbua2008) and Chang (Reference Chang2009) on reading-while-listening can be considered important triggers in the study of multimodal input for L2 learning. After 10 years of intensive research, multimodal input has now earned its place in both SLA (Second Language Acquisition) and CALL (Computer Assisted Language Learning) research and takes up a central position on the international research agenda. This is not only evident from the current special issue, but also from previous initiatives such as the special issue on video and language learning for the Language Learning Journal (Montero Perez & Rodgers, Reference Montero Perez and Rodgers2019) and several symposia on multimodal input and language learning at international conferences (e.g., EuroSLA, 2018 and SLRF, 2018).
An analysis of the studies that have been published during the last decade shows that there is a wide range of possibilities to operationalize multimodal input for L2 learners. First of all, we notice that an increasing number of studies on multimodal input is based on audio-visual input (e.g., Rodgers & Webb, Reference Rodgers and Webb2017) such as documentaries, TV series, films, or news clips that presents a combination of visuals (i.e., dynamic images) and sound. The bulk of studies has, however, looked into the role of audio-visual input enhanced with on-screen text. Previous research has shown that on-screen text can come in many different forms such as subtitles in the L1 (e.g., Peters et al., Reference Peters, Heynen and Puimège2016), subtitles in the L2 or captions (see Montero Perez et al., Reference Montero Perez, Van Den Noortgate and Desmet2013 for a meta-analysis), or other types of captions such as captions with enhanced target words or grammatical constructions (e.g., Cintrón-Valentín et al., Reference Cintrón-Valentín, García-Amaya and Ellis2019), captions with glosses (e.g., Montero Perez et al., Reference Montero Perez, Peters and Desmet2018), or keyword captions (e.g., Montero Perez et al., Reference Montero Perez, Peters, Clarebout and Desmet2014). Written text can also be made multimodal by adding, for instance, images and/or the accompanying audio, as is the case for reading-while-listening activities (e.g., Serrano & Pellicer-Sánchez, Reference Serrano and Pellicer-Sánchez2019; Tragant & Pellicer-Sánchez, Reference Tragant and Pellicer-Sánchez2019; Webb & Chang, Reference Webb and Chang2015). Another option consists in enhancing written text with hyperlinks that link words in the text to multimodal annotations or glosses that contain word-meaning information in the form of text, sound, and/or pictures (e.g., Boers et al., Reference Boers, Warren, Grimshaw and Siyanova-Chanturia2017; Warren et al., Reference Warren, Boers, Grimshaw and Siyanova-Chanturia2018). These types of multimodal input have been studied from different perspectives and may influence not only L2 comprehension but also other aspects of L2 development such as L2 word learning or speech perception (see Montero Perez, Reference Montero Perez, Arnold and Ducate2019b, for a comprehensive review).
From a theoretical point of view, SLA research on multimodal input has often been linked with Mayer’s (Reference Mayer2001) Cognitive Theory of Multimedia Learning that builds on Paivio’s dual-coding theory (1986). The central idea behind this theory is that human cognition presents separate channels for the processing of visual and verbal information. Even though these channels work separately, multimedia learning theory posits that learning outcomes can be enhanced when both channels are activated simultaneously. This is at the basis of the multimedia learning principle, which holds that “people learn better from words and pictures than from words alone” (Mayer, Reference Mayer2001, p. 63). Numerous studies have empirically confirmed this principle in the context of first language science learning (see Mayer, Reference Mayer2005 for an overview). Yet, it can also be seen that an increasing number of studies in the field of L2 learning from multimodal input draw on principles of this theory to frame research questions or explain research findings (e.g., Rodgers, Reference Rodgers2018). It should, however, be noted that the cognitive theory of multimedia learning was originally not conceived as a theory of L2 learning from multimodal input. This implies that multimedia learning principles might be different for L1 learning contexts than for L2 learning contexts. As we see in the outcomes of previous research as well as in the results of studies in this special issue, this is clearly the case for the redundancy principle, which holds that “students learn better from animation and narration than from animation, narration, and text” (Mayer, Reference Mayer2001, p. 147). Yet, numerous studies have shown that combining audio or audio-visual input with accompanying (redundant) text results in better comprehension of the L2 input as well as in more linguistic benefits such as vocabulary learning. This is due to the fact that the presence of written input helps learners to decode and segment the speech stream and supports the development of learners’ bottom-up listening processes (Charles & Trenkic, Reference Charles, Trenkic, Gambier, Caimi and Mariotti2015). The more a learner’s decoding skills are developed, the more time learners have to interpret the message or to notice unknown words. As argued by Plass and Jones (Reference Plass, Jones and Mayer2005, p. 480), “[R]eading and listening are two competencies that both need to be developed, and in many cases one is used as input enhancement for the other.”
Other multimedia learning principles, however, seem to operate similarly in L1 and L2 learning contexts. Recent research into the role of imagery for L2 learning from audio-visual input has, for instance, shown that the spatial and temporal contiguity principle, which holds that “students learn better when corresponding words and pictures are presented near rather than far from each other” (Mayer, Reference Mayer2001, p. 81), that is when they are “presented simultaneously rather than successively” (ibid., p. 96), can play an important role when selecting video materials for L2 learners. In this respect, a corpus-based study (Rodgers, Reference Rodgers2018) revealed that 65% of potential target words (i.e., words that might be learned) in a documentary appeared simultaneously with the image, which might stimulate the learning of those words. This hypothesis was empirically investigated by Peters (Reference Peters2019, p. 1023) who reported that “words with on-screen imagery are almost three times more likely to be picked up incidentally than words without imagery” when the image appeared within a 5-second time frame before or after the presentation of the word in the auditory trace. This study thus provides empirical support for the spatial and temporal contiguity principle as well as for the multimedia learning principle. More importantly, this study also reveals that learners are able to use different sources of input (images, on-screen text, audio) offered by multimodal input in their meaning-making process, which may in turn lead to L2 learning gains.
EXPLAINING THE IMPORTANCE OF MULTIMODAL INPUT
There are at least three reasons that may explain the growing importance of multimodal input for SLA researchers and practitioners.
A first reason undoubtedly lies in the overall availability and easy accessibility of multimodal L2 input. Webb (Reference Webb, Nunan and Richards2015) argued that the “greatest value of television for language learning might be its potential to provide large amounts of L2 spoken input” (p. 159). In addition, the quickly growing market of streaming services such as Netflix, HBO, or Amazon Prime Video; the availability of other resources such as YouTube; and an increasing number of TV channel websites offer enormous selections of TV programs, series, and films in different languages. We also see that the aforementioned services often come with numerous options to modify the input (see Chapelle, Reference Chapelle2003 on input modification), which consist in the addition of information such as on-screen text to stimulate comprehension and language learning. One example is the Google Chrome extension “language learning with Netflix,” which offers language support in the form of, for instance, subtitles in the target language. Words in those subtitles are clickable in the sense that they are linked to a dictionary that provides learners with access to word-meaning information. It also enables learners to create their own dictionary based on the series and programs they watched. Together, these tools and services highlight the role of technological advances. Indeed, new technologies have taken the realization of multimodal input to the next level by offering numerous options that were not available through more traditional (e.g., paper-based) resources.
The overall availability of L2 multimodal input is inextricably linked to findings on the role of out-of-class exposure for L2 development. In this respect, a number of studies on the role of English language exposure outside the classroom reported that watching TV was one of the most important out-of-class activities (Kuppens, Reference Kuppens2010; Lindgren & Muñoz, Reference Lindgren and Muñoz2013; Peters, Reference Peters2018). In addition, it was found that watching foreign language movies or TV programs outside of the classroom was positively related to aspects of L2 development. Studies showed, for instance, that learners who often watched English language TV programs or movies had a larger receptive (Peters, Reference Peters2018; Puimège & Peters, Reference Puimège and Peters2019) or productive (Bollansée et al., Reference Bollansée, Puimège, Peters, Werner and Teggein press) vocabulary than learners who were less frequently exposed to this type of L2 input. Together, these findings indicate that multimodal input is not only easily accessible but frequent exposure to this type of input may also benefit learners’ L2 development. In addition, it provides an interesting approach to stimulate contact with the L2 in contexts where exposure to the L2 is limited to the classroom.
A second reason why multimodal input has become an important research topic is theoretical in nature. Research into the effectiveness of different types of multimodal input is part of a larger research track that studies the impact of input and its characteristics on L2 acquisition from a cognitively oriented perspective. As can be seen in this special issue, multimodal input provides numerous options to study SLA constructs such as salience, noticing, and input enhancement. Studies have, for instance, used eye-tracking to investigate whether input enhancement in the form of visually enhanced target items or salient grammatical structures draws learners’ attention to target items (Lee & Révész, Reference Lee and Révész2018; Montero Perez et al., Reference Montero Perez, Peters and Desmet2015). Multimodal input also offers multiple options to study learners’ attention allocation to different types of materials such as reading-while-listening (Serrano & Pellicer-Sánchez, Reference Serrano and Pellicer-Sánchez2019) or captioned video (Montero Perez, Reference Montero Perez2019a) to investigate how learners divide their attention over different input modes (e.g., text and images) and how learners’ attention allocation relates to linguistic outcomes (see Pellicer-Sánchez et al., Reference Pellicer-Sánchez, Tragant, Conklin, Rodgers, Serrano and Llanes2020).
Finally, the third reason is a pedagogical one and concerns the fact that multimodal input may provide language teachers with effective options for classroom practice, which go beyond typical comprehension activities. In instructed SLA contexts (i.e., contexts where teaching time is typically limited to 2 or 3 hours of instruction per week), however, extensive exposure to multimodal input is not always possible. Yet, research has also highlighted the value of short clips for stimulating various aspects of L2 learning. Classroom-based use of multimodal input could, for instance, focus on the training of listening fluency. To become a fluent listener, researchers have argued that we need a successful interaction of learners’ bottom-up (i.e., decoding) and top-down (i.e., learners’ conceptual knowledge) processes (Rost, Reference Rost, Muller, Adamson, Brown and Herder2014). While top-down processes might be stimulated by the presence of, for instance, imagery and related storylines (Rodgers, 2013), multimodal input also presents numerous ways to train bottom-up listening skills. As shown by Wisnieska and Mora (Reference Wisniewska and Mora2020) as well as by previous research (Charles & Trenkic, Reference Charles, Trenkic, Gambier, Caimi and Mariotti2015), frequent exposure to captioned audiovisual input can be considered an effective technique to stimulate learners’ speech segmentation and decoding skills in that it does not only aid their processing of the captioned input but also of newly encountered input (Charles & Trenkic, Reference Charles, Trenkic, Gambier, Caimi and Mariotti2015). By providing a real-time written account of the auditory trace, we cannot only reduce learners’ cognitive processing load but also give them more time to process and interpret the meaning. This may in turn also lead to better noticing of new words in the input (Winke et al., Reference Winke, Gass and Sydorenko2010) or stimulate other aspects of L2 learning such as grammar (Cintrón-Valentín et al., Reference Cintrón-Valentín, García-Amaya and Ellis2019). In addition, by processing multiple input modes (cf. multimedia learning principle), we may not only boost L2 learning but also learners’ retention of the information.
CONTRIBUTION OF THIS SPECIAL ISSUE
This special issue illustrates that multimodal input can be operationalized in many different ways. Studies have investigated the value of audio-visual input (Feng & Webb, Reference Feng and Webb2020; Puimège & Peters, Reference Puimège and Peters2020), audio-visual input with subtitles in the L1 (Pujadas & Muñoz, Reference Pujadas and Muñoz2020), subtitles in the L2 (Lee & Révész, Reference Lee and Révész2020; Pujadas & Muñoz, Reference Pujadas and Muñoz2020), and L2 subtitles with enhanced (i.e., highlighted) target constructions (Lee & Révész, Reference Lee and Révész2020). It also offers a study on audio-assisted reading activities, that is, reading while listening (Pellicer-Sánchez et al., Reference Pellicer-Sánchez, Tragant, Conklin, Rodgers, Serrano and Llanes2020). In what follows, I discuss some of the main implicationsFootnote 1 of the aforementioned studies for theory, methodology, and pedagogy.
THEORETICAL CONTRIBUTION
A first theoretical contribution of this special issue is that it investigates aspects of L2 learning that have, so far, received little attention. More particularly, it expands upon existing studies that focused mainly on the learning of single words through multimodal input by looking into the potential of multimodal input for other aspects of L2 learning such as grammar (Lee & Révész, Reference Lee and Révész2020), pronunciation (Wisnieska & Mora, Reference Wisniewska and Mora2020), and formulaic sequences (Puimège & Peters, Reference Puimège and Peters2020). In addition, this special issue provides an answer to the long-standing call for more longitudinal research (Vanderplank, Reference Vanderplank2016) on the effects of multimodal input. In Pujadas and Muñoz (Reference Pujadas and Muñoz2020) the effects of watching a TV series on L2 learners’ comprehension were investigated in an 8-month extensive viewing intervention program (or 515 minutes of total exposure). An 8-week intervention program (or 320 minutes of exposure to audiovisual input) was used in Wisnieska and Mora (Reference Wisniewska and Mora2020) to investigate its effects on L2 speech segmentation, processing, and pronunciation.
Second, this special issue corroborates the beneficial impact of captioned video on L2 learning (Lee & Révész, Reference Lee and Révész2020; Wisnieska & Mora, Reference Wisniewska and Mora2020) and provides additional evidence to reject the redundancy principle of the multimedia learning theory in the context of L2 learning. In addition, it sheds new light on the impact of L2 text on learners’ auditory processing of the input. While Diao et al. (2007) reported negative effects of captioned audio on L2 listeners’ auditory processing, Wisnieska and Mora (Reference Wisniewska and Mora2020) provide further evidence that both uncaptioned and captioned video can facilitate L2 learners’ speech segmentation and processing.Footnote 2
A third theoretical merit of this special issue is that it contributes to our understanding of incidental learning processes that may occur while processing multimodal input. Feng and Webb (Reference Feng and Webb2020) do so by comparing the potential of written, auditory, and audiovisual input to determine the relative value of these three input modes for incidental vocabulary learning (of single words). Puimège and Peters (Reference Puimège and Peters2020) investigated whether and to what extent incidental learning from audio-visual input is mediated by a combination of learner-related factors (i.e., learners’ prior vocabulary knowledge) and factors related to the target formulaic sequences (i.e., collocate-node relationship, congruency, transparency, and statistical measures). They also included participants’ prior knowledge of target items at the level of meaning recall, which was found to predict learners’ uptake of word form. By including these factors, this study does not only provide information on the mediating role of learner- and word-related factors for incidental learning from multimodal input but they also provide evidence for the incremental nature of (incidental) vocabulary learning. In addition, these findings highlight the value of exposure to audio-visual input as a way to consolidate and refine learners’ L2 vocabulary knowledge.
A fourth theoretical contribution concerns the fact that this special issue adds to the much discussed role of attention for L2 learning from input and the relationship between attention and aspects of L2 learning. The study by Pellicer-Sánchez et al. (Reference Pellicer-Sánchez, Tragant, Conklin, Rodgers, Serrano and Llanes2020) contributes to this discussion by investigating how learners process multimodal input. More particularly, they examine whether and to what extent the presence of audio influences young English as a second language (ESL) learners’ reading (as measured in terms of eye movements) of an illustrated graded reader by comparing a reading-only and a reading-while-listening group. In addition, by combining data from eye-movement analyses and a comprehension test, they provide information on whether and how attention on text and images is related to learners’ scores on the comprehension test. Lee and Révész (Reference Lee and Révész2020) expand on previous findings on the role of input enhancement (i.e., perceptually salient structures) by investigating its role in captioned audiovisual input. In their study, input enhancement was operationalized by highlighting target grammatical constructions (i.e., verbs in the present perfect and past simple) in the captions to direct learners’ attention to these constructions. They found that textual enhancement does not only have the potential to draw L2 learners’ attention, as measured in terms of learners’ eye fixations, but they also reported a positive relationship between learners’ eye movements and their learning gains.
METHODOLOGICAL CONTRIBUTION
At the methodological level, I believe this special issue highlights, at least, two main features that future studies into the role of multimodal input for L2 learning might need to consider. First, it illustrates how different data-collection techniques ranging from paper-and-pencil tests (e.g., comprehension or vocabulary tests), interviews, or questionnaires (e.g., on language exposure) to more recent techniques such as eye-tracking can be used to investigate different components of L2 learning from multimodal input. More particularly, the special issue provides interesting examples of fine-grained (Pujadas & Muñoz, Reference Pujadas and Muñoz2020) and multimodal comprehension tests (Pellicer-Sánchez et al., Reference Pellicer-Sánchez, Tragant, Conklin, Rodgers, Serrano and Llanes2020). It also highlights the role of tests that target different aspects of L2 knowledge development (e.g., Wisnieska & Mora, Reference Wisniewska and Mora2020) and that reveal partial learning gains (e.g., Feng & Webb, Reference Feng and Webb2020; Puimège & Peters, Reference Puimège and Peters2020). We also find two examples of how eye-tracking can be used to study multimodal input and which types of eye-tracking metrics we might need to consider (Lee & Révész, Reference Lee and Révész2020; Pellicer-Sánchez et al., Reference Pellicer-Sánchez, Tragant, Conklin, Rodgers, Serrano and Llanes2020). While eye-tracking is a particularly interesting technique to study multimodal input because it provides an online unobtrusive account of learners’ attention allocation, the multimodal character of the input also has a number of implications for data collection procedures and analyses. For a detailed description on eye-tracking methodology in studies on multimodal input, we refer to Godfroid (Reference Godfroid2019) and Conklin et al. (Reference Conklin, Pellicer-Sanchez and Carrol2018).
Second, this special issue as well as other recently published studies demonstrate that L2 research into multimodal input should take into account a variety of factors in the study design. With regard to learner-related factors, it has been found that proficiency level (e.g., Suárez & Gesa, Reference Suárez and Gesa2019), prior vocabulary knowledge (e.g., Feng & Webb, Reference Feng and Webb2020; Peters & Webb, Reference Peters and Webb2018; Puimège & Peters, Reference Puimège and Peters2020), age (Muñoz, Reference Muñoz2017), or working memory (e.g., Gass et al., Reference Gass, Winke, Isbell and Ahn2019; Montero Perez, Reference Montero Perez2020) may impact learning gains and the processing of multimodal input. Studies also indicate that including factors related to the target items (e.g., frequency of occurrence, part of speech, visual representation) or to the input (e.g., lexical coverage of TV episodes) is not only important to control for these variables but it also indicates whether these factors influence the learning process from multimodal input (Peters & Webb, Reference Peters and Webb2018; Puimège & Peters, Reference Puimège and Peters2020). It is therefore not only crucial to include a detailed description of the multimodal input, individual differences, and the target items in the study methodology but also to use appropriate statistical techniques that allow for the inclusion of learner- and item-related factors such as logistic regression or mixed-effects models (e.g., Puimège & Peters, Reference Puimège and Peters2020; Pujadas & Muñoz, Reference Pujadas and Muñoz2020).
PEDAGOGICAL CONTRIBUTION
This special issue presents research findings on the effectiveness of different types of multimodal input (e.g., audiovisual input, reading while listening) for L2 teaching and learning, how L2 learners process multimodal information, and whether this relates to learning. It therefore offers a number of pedagogical recommendations that language learners and teachers might want to consider.
With regard to the use of audio-visual input (without on-screen text) for L2 vocabulary learning, this special issue shows that it does not only stimulate the learning of single words (Feng & Webb, Reference Feng and Webb2020) but also of formulaic sequences (Puimège & Peters, Reference Puimège and Peters2020). In addition, it shows that the uptake of new vocabulary from audio-visual input may be comparable to other types of input such as written text and audio (Feng & Webb, Reference Feng and Webb2020). These findings are encouraging: they show that more extensive exposure to video can yield L2 vocabulary development and confirm the role of audiovisual materials as a valuable means of increasing L2 learners’ contact with the L2 in limited-input instructional settings. In addition, watching video can be considered a motivating alternative for L2 learners who are less inclined to use L2 written or listening input in out-of-class contexts (Peters, Reference Peters2018).
Because captioned video does not only seem to aid speech decoding and pronunciation but also facilitates other aspects of L2 development such as vocabulary and grammar (Lee & Révész, Reference Lee and Révész2020), language teachers should be encouraged to further explore the value of this type of multimodal input in their teaching and stimulate students to engage with those materials as out-of-class activity. In addition, they might also consider the use of other types of captions such as captions with enhanced target structures if the aim is to draw learners’ attention to specific linguistic features (e.g., grammatical structures) and stimulate their learning of those structures (Lee & Révész, Reference Lee and Révész2020).
Multimodal input is not only beneficial for intermediate to advanced students (e.g., Lee & Révész, Reference Lee and Révész2020; Puimège & Peters, Reference Puimège and Peters2020; Wisnieska & Mora, Reference Wisniewska and Mora2020) but can also be used with young and adolescent beginning L2 learners (Pellicer-Sánchez et al., Reference Pellicer-Sánchez, Tragant, Conklin, Rodgers, Serrano and Llanes2020; Pujadas & Muñoz, Reference Pujadas and Muñoz2020). Both reading while listening (with supportive images) as well as videos with subtitles in the L1 have the potential to motivate beginning L2 learners to engage with this type of L2 input and facilitates their comprehension of those materials. Pedagogical approaches that stimulate the use of these materials early on in the learning process should therefore be promoted, both inside and outside the classroom (see Tragant et al., Reference Tragant, Muñoz and Spada2016, for an example).
Finally, it is worth noting that the study by Lee and Révész (Reference Lee and Révész2020) differs from many other studies in the field by implementing videos in an actual pedagogical approach, namely task-based language teaching. One of the merits of this study is that it clearly illustrates how short videos can be used in meaningful input-based tasks that are effective in stimulating L2 grammar learning.
Drawing on the previously mentioned implications and contributions of this special issue, I hope that researchers are inspired to conduct further research on the role of multimodal input for L2 learning. This will undoubtedly continue to refine our understanding of the numerous interesting findings and research questions put forward by the studies in this special issue.
You have
Access
INTRODUCTION
The studies in this special issue, Language Learning from Multimodal Input, investigate the potential of multimodal input for language learning from different angles. They focus on different types of linguistic outcomes, such as grammar, vocabulary (both single words and formulaic sequences), pronunciation, and comprehension but also on L2 learners’ processing of multimodal input. In doing so, these studies examine research questions that have, so far, received little attention in the research literature and, thus, provide an important contribution to the field.
My commentary on this special issue consists of three parts. First, I discuss aspects related to the multimodality of input and the potential role of the multimedia learning theory (Mayer, Reference Mayer2001) in studies on multimodal L2 input. Next, I put forward three main reasons that may explain the importance of multimodal input for SLA researchers. I conclude this commentary by discussing the contribution of this special issue to theory, methodology, and pedagogy.
DEFINING MULTIMODAL INPUT
As language users, we are constantly surrounded by multimodal input (see Kress & Van Leeuwen, Reference Kress and Van Leeuwen2001 on multimodality). For instance, when we watch TV, we deal with incoming sound and images. In everyday conversations, we process sound, gestures, body language, and so forth. Being able to extract meaning from multimodal input is thus a crucial component of everyday life. Drawing on the multimodality of everyday life, the Douglas Fir Group (2016) has defined language learning as semiotic learning, which means that we can rely on a wide range of semiotic resources in our L2 meaning-making and learning processes, such as nonverbal information (e.g., facial expressions, body language), but also textual, graphic, or auditory resources. Language learning is thus necessarily multimodal learning.
During the last decade, we have witnessed a remarkable number of studies on the role of multimodal input for aspects of L2 acquisition. Even though studies already explored the potential of multimodal input in the nineties and the early 2000s (see Vanderplank, Reference Vanderplank2010 for a comprehensive review), we can clearly notice a renewed and reinforced interest into the role of multimodal input around 2009–2010. First of all, there is the publication of a number of corpus-based studies (e.g., Webb & Rodgers, Reference Webb and Rodgers2009; Webb, Reference Webb2010a, Reference Webb2010b) that highlight the potential of TV as an important source of L2 input by investigating the vocabulary demands of television programs such as TV series and films to determine target vocabulary sizes needed for adequate comprehension of those programs. Second, a number of empirical studies such as the ones carried out by Winke et al. (Reference Winke, Gass and Sydorenko2010) and Sydorenko (Reference Sydorenko2010) on captioned video or Brown et al. (Reference Brown, Waring and Donkaewbua2008) and Chang (Reference Chang2009) on reading-while-listening can be considered important triggers in the study of multimodal input for L2 learning. After 10 years of intensive research, multimodal input has now earned its place in both SLA (Second Language Acquisition) and CALL (Computer Assisted Language Learning) research and takes up a central position on the international research agenda. This is not only evident from the current special issue, but also from previous initiatives such as the special issue on video and language learning for the Language Learning Journal (Montero Perez & Rodgers, Reference Montero Perez and Rodgers2019) and several symposia on multimodal input and language learning at international conferences (e.g., EuroSLA, 2018 and SLRF, 2018).
An analysis of the studies that have been published during the last decade shows that there is a wide range of possibilities to operationalize multimodal input for L2 learners. First of all, we notice that an increasing number of studies on multimodal input is based on audio-visual input (e.g., Rodgers & Webb, Reference Rodgers and Webb2017) such as documentaries, TV series, films, or news clips that presents a combination of visuals (i.e., dynamic images) and sound. The bulk of studies has, however, looked into the role of audio-visual input enhanced with on-screen text. Previous research has shown that on-screen text can come in many different forms such as subtitles in the L1 (e.g., Peters et al., Reference Peters, Heynen and Puimège2016), subtitles in the L2 or captions (see Montero Perez et al., Reference Montero Perez, Van Den Noortgate and Desmet2013 for a meta-analysis), or other types of captions such as captions with enhanced target words or grammatical constructions (e.g., Cintrón-Valentín et al., Reference Cintrón-Valentín, García-Amaya and Ellis2019), captions with glosses (e.g., Montero Perez et al., Reference Montero Perez, Peters and Desmet2018), or keyword captions (e.g., Montero Perez et al., Reference Montero Perez, Peters, Clarebout and Desmet2014). Written text can also be made multimodal by adding, for instance, images and/or the accompanying audio, as is the case for reading-while-listening activities (e.g., Serrano & Pellicer-Sánchez, Reference Serrano and Pellicer-Sánchez2019; Tragant & Pellicer-Sánchez, Reference Tragant and Pellicer-Sánchez2019; Webb & Chang, Reference Webb and Chang2015). Another option consists in enhancing written text with hyperlinks that link words in the text to multimodal annotations or glosses that contain word-meaning information in the form of text, sound, and/or pictures (e.g., Boers et al., Reference Boers, Warren, Grimshaw and Siyanova-Chanturia2017; Warren et al., Reference Warren, Boers, Grimshaw and Siyanova-Chanturia2018). These types of multimodal input have been studied from different perspectives and may influence not only L2 comprehension but also other aspects of L2 development such as L2 word learning or speech perception (see Montero Perez, Reference Montero Perez, Arnold and Ducate2019b, for a comprehensive review).
From a theoretical point of view, SLA research on multimodal input has often been linked with Mayer’s (Reference Mayer2001) Cognitive Theory of Multimedia Learning that builds on Paivio’s dual-coding theory (1986). The central idea behind this theory is that human cognition presents separate channels for the processing of visual and verbal information. Even though these channels work separately, multimedia learning theory posits that learning outcomes can be enhanced when both channels are activated simultaneously. This is at the basis of the multimedia learning principle, which holds that “people learn better from words and pictures than from words alone” (Mayer, Reference Mayer2001, p. 63). Numerous studies have empirically confirmed this principle in the context of first language science learning (see Mayer, Reference Mayer2005 for an overview). Yet, it can also be seen that an increasing number of studies in the field of L2 learning from multimodal input draw on principles of this theory to frame research questions or explain research findings (e.g., Rodgers, Reference Rodgers2018). It should, however, be noted that the cognitive theory of multimedia learning was originally not conceived as a theory of L2 learning from multimodal input. This implies that multimedia learning principles might be different for L1 learning contexts than for L2 learning contexts. As we see in the outcomes of previous research as well as in the results of studies in this special issue, this is clearly the case for the redundancy principle, which holds that “students learn better from animation and narration than from animation, narration, and text” (Mayer, Reference Mayer2001, p. 147). Yet, numerous studies have shown that combining audio or audio-visual input with accompanying (redundant) text results in better comprehension of the L2 input as well as in more linguistic benefits such as vocabulary learning. This is due to the fact that the presence of written input helps learners to decode and segment the speech stream and supports the development of learners’ bottom-up listening processes (Charles & Trenkic, Reference Charles, Trenkic, Gambier, Caimi and Mariotti2015). The more a learner’s decoding skills are developed, the more time learners have to interpret the message or to notice unknown words. As argued by Plass and Jones (Reference Plass, Jones and Mayer2005, p. 480), “[R]eading and listening are two competencies that both need to be developed, and in many cases one is used as input enhancement for the other.”
Other multimedia learning principles, however, seem to operate similarly in L1 and L2 learning contexts. Recent research into the role of imagery for L2 learning from audio-visual input has, for instance, shown that the spatial and temporal contiguity principle, which holds that “students learn better when corresponding words and pictures are presented near rather than far from each other” (Mayer, Reference Mayer2001, p. 81), that is when they are “presented simultaneously rather than successively” (ibid., p. 96), can play an important role when selecting video materials for L2 learners. In this respect, a corpus-based study (Rodgers, Reference Rodgers2018) revealed that 65% of potential target words (i.e., words that might be learned) in a documentary appeared simultaneously with the image, which might stimulate the learning of those words. This hypothesis was empirically investigated by Peters (Reference Peters2019, p. 1023) who reported that “words with on-screen imagery are almost three times more likely to be picked up incidentally than words without imagery” when the image appeared within a 5-second time frame before or after the presentation of the word in the auditory trace. This study thus provides empirical support for the spatial and temporal contiguity principle as well as for the multimedia learning principle. More importantly, this study also reveals that learners are able to use different sources of input (images, on-screen text, audio) offered by multimodal input in their meaning-making process, which may in turn lead to L2 learning gains.
EXPLAINING THE IMPORTANCE OF MULTIMODAL INPUT
There are at least three reasons that may explain the growing importance of multimodal input for SLA researchers and practitioners.
A first reason undoubtedly lies in the overall availability and easy accessibility of multimodal L2 input. Webb (Reference Webb, Nunan and Richards2015) argued that the “greatest value of television for language learning might be its potential to provide large amounts of L2 spoken input” (p. 159). In addition, the quickly growing market of streaming services such as Netflix, HBO, or Amazon Prime Video; the availability of other resources such as YouTube; and an increasing number of TV channel websites offer enormous selections of TV programs, series, and films in different languages. We also see that the aforementioned services often come with numerous options to modify the input (see Chapelle, Reference Chapelle2003 on input modification), which consist in the addition of information such as on-screen text to stimulate comprehension and language learning. One example is the Google Chrome extension “language learning with Netflix,” which offers language support in the form of, for instance, subtitles in the target language. Words in those subtitles are clickable in the sense that they are linked to a dictionary that provides learners with access to word-meaning information. It also enables learners to create their own dictionary based on the series and programs they watched. Together, these tools and services highlight the role of technological advances. Indeed, new technologies have taken the realization of multimodal input to the next level by offering numerous options that were not available through more traditional (e.g., paper-based) resources.
The overall availability of L2 multimodal input is inextricably linked to findings on the role of out-of-class exposure for L2 development. In this respect, a number of studies on the role of English language exposure outside the classroom reported that watching TV was one of the most important out-of-class activities (Kuppens, Reference Kuppens2010; Lindgren & Muñoz, Reference Lindgren and Muñoz2013; Peters, Reference Peters2018). In addition, it was found that watching foreign language movies or TV programs outside of the classroom was positively related to aspects of L2 development. Studies showed, for instance, that learners who often watched English language TV programs or movies had a larger receptive (Peters, Reference Peters2018; Puimège & Peters, Reference Puimège and Peters2019) or productive (Bollansée et al., Reference Bollansée, Puimège, Peters, Werner and Teggein press) vocabulary than learners who were less frequently exposed to this type of L2 input. Together, these findings indicate that multimodal input is not only easily accessible but frequent exposure to this type of input may also benefit learners’ L2 development. In addition, it provides an interesting approach to stimulate contact with the L2 in contexts where exposure to the L2 is limited to the classroom.
A second reason why multimodal input has become an important research topic is theoretical in nature. Research into the effectiveness of different types of multimodal input is part of a larger research track that studies the impact of input and its characteristics on L2 acquisition from a cognitively oriented perspective. As can be seen in this special issue, multimodal input provides numerous options to study SLA constructs such as salience, noticing, and input enhancement. Studies have, for instance, used eye-tracking to investigate whether input enhancement in the form of visually enhanced target items or salient grammatical structures draws learners’ attention to target items (Lee & Révész, Reference Lee and Révész2018; Montero Perez et al., Reference Montero Perez, Peters and Desmet2015). Multimodal input also offers multiple options to study learners’ attention allocation to different types of materials such as reading-while-listening (Serrano & Pellicer-Sánchez, Reference Serrano and Pellicer-Sánchez2019) or captioned video (Montero Perez, Reference Montero Perez2019a) to investigate how learners divide their attention over different input modes (e.g., text and images) and how learners’ attention allocation relates to linguistic outcomes (see Pellicer-Sánchez et al., Reference Pellicer-Sánchez, Tragant, Conklin, Rodgers, Serrano and Llanes2020).
Finally, the third reason is a pedagogical one and concerns the fact that multimodal input may provide language teachers with effective options for classroom practice, which go beyond typical comprehension activities. In instructed SLA contexts (i.e., contexts where teaching time is typically limited to 2 or 3 hours of instruction per week), however, extensive exposure to multimodal input is not always possible. Yet, research has also highlighted the value of short clips for stimulating various aspects of L2 learning. Classroom-based use of multimodal input could, for instance, focus on the training of listening fluency. To become a fluent listener, researchers have argued that we need a successful interaction of learners’ bottom-up (i.e., decoding) and top-down (i.e., learners’ conceptual knowledge) processes (Rost, Reference Rost, Muller, Adamson, Brown and Herder2014). While top-down processes might be stimulated by the presence of, for instance, imagery and related storylines (Rodgers, 2013), multimodal input also presents numerous ways to train bottom-up listening skills. As shown by Wisnieska and Mora (Reference Wisniewska and Mora2020) as well as by previous research (Charles & Trenkic, Reference Charles, Trenkic, Gambier, Caimi and Mariotti2015), frequent exposure to captioned audiovisual input can be considered an effective technique to stimulate learners’ speech segmentation and decoding skills in that it does not only aid their processing of the captioned input but also of newly encountered input (Charles & Trenkic, Reference Charles, Trenkic, Gambier, Caimi and Mariotti2015). By providing a real-time written account of the auditory trace, we cannot only reduce learners’ cognitive processing load but also give them more time to process and interpret the meaning. This may in turn also lead to better noticing of new words in the input (Winke et al., Reference Winke, Gass and Sydorenko2010) or stimulate other aspects of L2 learning such as grammar (Cintrón-Valentín et al., Reference Cintrón-Valentín, García-Amaya and Ellis2019). In addition, by processing multiple input modes (cf. multimedia learning principle), we may not only boost L2 learning but also learners’ retention of the information.
CONTRIBUTION OF THIS SPECIAL ISSUE
This special issue illustrates that multimodal input can be operationalized in many different ways. Studies have investigated the value of audio-visual input (Feng & Webb, Reference Feng and Webb2020; Puimège & Peters, Reference Puimège and Peters2020), audio-visual input with subtitles in the L1 (Pujadas & Muñoz, Reference Pujadas and Muñoz2020), subtitles in the L2 (Lee & Révész, Reference Lee and Révész2020; Pujadas & Muñoz, Reference Pujadas and Muñoz2020), and L2 subtitles with enhanced (i.e., highlighted) target constructions (Lee & Révész, Reference Lee and Révész2020). It also offers a study on audio-assisted reading activities, that is, reading while listening (Pellicer-Sánchez et al., Reference Pellicer-Sánchez, Tragant, Conklin, Rodgers, Serrano and Llanes2020). In what follows, I discuss some of the main implicationsFootnote 1 of the aforementioned studies for theory, methodology, and pedagogy.
THEORETICAL CONTRIBUTION
A first theoretical contribution of this special issue is that it investigates aspects of L2 learning that have, so far, received little attention. More particularly, it expands upon existing studies that focused mainly on the learning of single words through multimodal input by looking into the potential of multimodal input for other aspects of L2 learning such as grammar (Lee & Révész, Reference Lee and Révész2020), pronunciation (Wisnieska & Mora, Reference Wisniewska and Mora2020), and formulaic sequences (Puimège & Peters, Reference Puimège and Peters2020). In addition, this special issue provides an answer to the long-standing call for more longitudinal research (Vanderplank, Reference Vanderplank2016) on the effects of multimodal input. In Pujadas and Muñoz (Reference Pujadas and Muñoz2020) the effects of watching a TV series on L2 learners’ comprehension were investigated in an 8-month extensive viewing intervention program (or 515 minutes of total exposure). An 8-week intervention program (or 320 minutes of exposure to audiovisual input) was used in Wisnieska and Mora (Reference Wisniewska and Mora2020) to investigate its effects on L2 speech segmentation, processing, and pronunciation.
Second, this special issue corroborates the beneficial impact of captioned video on L2 learning (Lee & Révész, Reference Lee and Révész2020; Wisnieska & Mora, Reference Wisniewska and Mora2020) and provides additional evidence to reject the redundancy principle of the multimedia learning theory in the context of L2 learning. In addition, it sheds new light on the impact of L2 text on learners’ auditory processing of the input. While Diao et al. (2007) reported negative effects of captioned audio on L2 listeners’ auditory processing, Wisnieska and Mora (Reference Wisniewska and Mora2020) provide further evidence that both uncaptioned and captioned video can facilitate L2 learners’ speech segmentation and processing.Footnote 2
A third theoretical merit of this special issue is that it contributes to our understanding of incidental learning processes that may occur while processing multimodal input. Feng and Webb (Reference Feng and Webb2020) do so by comparing the potential of written, auditory, and audiovisual input to determine the relative value of these three input modes for incidental vocabulary learning (of single words). Puimège and Peters (Reference Puimège and Peters2020) investigated whether and to what extent incidental learning from audio-visual input is mediated by a combination of learner-related factors (i.e., learners’ prior vocabulary knowledge) and factors related to the target formulaic sequences (i.e., collocate-node relationship, congruency, transparency, and statistical measures). They also included participants’ prior knowledge of target items at the level of meaning recall, which was found to predict learners’ uptake of word form. By including these factors, this study does not only provide information on the mediating role of learner- and word-related factors for incidental learning from multimodal input but they also provide evidence for the incremental nature of (incidental) vocabulary learning. In addition, these findings highlight the value of exposure to audio-visual input as a way to consolidate and refine learners’ L2 vocabulary knowledge.
A fourth theoretical contribution concerns the fact that this special issue adds to the much discussed role of attention for L2 learning from input and the relationship between attention and aspects of L2 learning. The study by Pellicer-Sánchez et al. (Reference Pellicer-Sánchez, Tragant, Conklin, Rodgers, Serrano and Llanes2020) contributes to this discussion by investigating how learners process multimodal input. More particularly, they examine whether and to what extent the presence of audio influences young English as a second language (ESL) learners’ reading (as measured in terms of eye movements) of an illustrated graded reader by comparing a reading-only and a reading-while-listening group. In addition, by combining data from eye-movement analyses and a comprehension test, they provide information on whether and how attention on text and images is related to learners’ scores on the comprehension test. Lee and Révész (Reference Lee and Révész2020) expand on previous findings on the role of input enhancement (i.e., perceptually salient structures) by investigating its role in captioned audiovisual input. In their study, input enhancement was operationalized by highlighting target grammatical constructions (i.e., verbs in the present perfect and past simple) in the captions to direct learners’ attention to these constructions. They found that textual enhancement does not only have the potential to draw L2 learners’ attention, as measured in terms of learners’ eye fixations, but they also reported a positive relationship between learners’ eye movements and their learning gains.
METHODOLOGICAL CONTRIBUTION
At the methodological level, I believe this special issue highlights, at least, two main features that future studies into the role of multimodal input for L2 learning might need to consider. First, it illustrates how different data-collection techniques ranging from paper-and-pencil tests (e.g., comprehension or vocabulary tests), interviews, or questionnaires (e.g., on language exposure) to more recent techniques such as eye-tracking can be used to investigate different components of L2 learning from multimodal input. More particularly, the special issue provides interesting examples of fine-grained (Pujadas & Muñoz, Reference Pujadas and Muñoz2020) and multimodal comprehension tests (Pellicer-Sánchez et al., Reference Pellicer-Sánchez, Tragant, Conklin, Rodgers, Serrano and Llanes2020). It also highlights the role of tests that target different aspects of L2 knowledge development (e.g., Wisnieska & Mora, Reference Wisniewska and Mora2020) and that reveal partial learning gains (e.g., Feng & Webb, Reference Feng and Webb2020; Puimège & Peters, Reference Puimège and Peters2020). We also find two examples of how eye-tracking can be used to study multimodal input and which types of eye-tracking metrics we might need to consider (Lee & Révész, Reference Lee and Révész2020; Pellicer-Sánchez et al., Reference Pellicer-Sánchez, Tragant, Conklin, Rodgers, Serrano and Llanes2020). While eye-tracking is a particularly interesting technique to study multimodal input because it provides an online unobtrusive account of learners’ attention allocation, the multimodal character of the input also has a number of implications for data collection procedures and analyses. For a detailed description on eye-tracking methodology in studies on multimodal input, we refer to Godfroid (Reference Godfroid2019) and Conklin et al. (Reference Conklin, Pellicer-Sanchez and Carrol2018).
Second, this special issue as well as other recently published studies demonstrate that L2 research into multimodal input should take into account a variety of factors in the study design. With regard to learner-related factors, it has been found that proficiency level (e.g., Suárez & Gesa, Reference Suárez and Gesa2019), prior vocabulary knowledge (e.g., Feng & Webb, Reference Feng and Webb2020; Peters & Webb, Reference Peters and Webb2018; Puimège & Peters, Reference Puimège and Peters2020), age (Muñoz, Reference Muñoz2017), or working memory (e.g., Gass et al., Reference Gass, Winke, Isbell and Ahn2019; Montero Perez, Reference Montero Perez2020) may impact learning gains and the processing of multimodal input. Studies also indicate that including factors related to the target items (e.g., frequency of occurrence, part of speech, visual representation) or to the input (e.g., lexical coverage of TV episodes) is not only important to control for these variables but it also indicates whether these factors influence the learning process from multimodal input (Peters & Webb, Reference Peters and Webb2018; Puimège & Peters, Reference Puimège and Peters2020). It is therefore not only crucial to include a detailed description of the multimodal input, individual differences, and the target items in the study methodology but also to use appropriate statistical techniques that allow for the inclusion of learner- and item-related factors such as logistic regression or mixed-effects models (e.g., Puimège & Peters, Reference Puimège and Peters2020; Pujadas & Muñoz, Reference Pujadas and Muñoz2020).
PEDAGOGICAL CONTRIBUTION
This special issue presents research findings on the effectiveness of different types of multimodal input (e.g., audiovisual input, reading while listening) for L2 teaching and learning, how L2 learners process multimodal information, and whether this relates to learning. It therefore offers a number of pedagogical recommendations that language learners and teachers might want to consider.
With regard to the use of audio-visual input (without on-screen text) for L2 vocabulary learning, this special issue shows that it does not only stimulate the learning of single words (Feng & Webb, Reference Feng and Webb2020) but also of formulaic sequences (Puimège & Peters, Reference Puimège and Peters2020). In addition, it shows that the uptake of new vocabulary from audio-visual input may be comparable to other types of input such as written text and audio (Feng & Webb, Reference Feng and Webb2020). These findings are encouraging: they show that more extensive exposure to video can yield L2 vocabulary development and confirm the role of audiovisual materials as a valuable means of increasing L2 learners’ contact with the L2 in limited-input instructional settings. In addition, watching video can be considered a motivating alternative for L2 learners who are less inclined to use L2 written or listening input in out-of-class contexts (Peters, Reference Peters2018).
Because captioned video does not only seem to aid speech decoding and pronunciation but also facilitates other aspects of L2 development such as vocabulary and grammar (Lee & Révész, Reference Lee and Révész2020), language teachers should be encouraged to further explore the value of this type of multimodal input in their teaching and stimulate students to engage with those materials as out-of-class activity. In addition, they might also consider the use of other types of captions such as captions with enhanced target structures if the aim is to draw learners’ attention to specific linguistic features (e.g., grammatical structures) and stimulate their learning of those structures (Lee & Révész, Reference Lee and Révész2020).
Multimodal input is not only beneficial for intermediate to advanced students (e.g., Lee & Révész, Reference Lee and Révész2020; Puimège & Peters, Reference Puimège and Peters2020; Wisnieska & Mora, Reference Wisniewska and Mora2020) but can also be used with young and adolescent beginning L2 learners (Pellicer-Sánchez et al., Reference Pellicer-Sánchez, Tragant, Conklin, Rodgers, Serrano and Llanes2020; Pujadas & Muñoz, Reference Pujadas and Muñoz2020). Both reading while listening (with supportive images) as well as videos with subtitles in the L1 have the potential to motivate beginning L2 learners to engage with this type of L2 input and facilitates their comprehension of those materials. Pedagogical approaches that stimulate the use of these materials early on in the learning process should therefore be promoted, both inside and outside the classroom (see Tragant et al., Reference Tragant, Muñoz and Spada2016, for an example).
Finally, it is worth noting that the study by Lee and Révész (Reference Lee and Révész2020) differs from many other studies in the field by implementing videos in an actual pedagogical approach, namely task-based language teaching. One of the merits of this study is that it clearly illustrates how short videos can be used in meaningful input-based tasks that are effective in stimulating L2 grammar learning.
Drawing on the previously mentioned implications and contributions of this special issue, I hope that researchers are inspired to conduct further research on the role of multimodal input for L2 learning. This will undoubtedly continue to refine our understanding of the numerous interesting findings and research questions put forward by the studies in this special issue.