MULTIMODAL INPUT

Drawing on Paivio’s (Reference Paivio1986) Dual Coding Theory, Mayer’s (Reference Mayer and Mayer2014) cognitive theory of multimedia learning states that learning is better when information is processed in spoken as well as written mode because learners make mental connections between the aural and visual information provided there is temporal proximity. Examples in the domain of language learning are storybooks with pictures read aloud (e.g., Tragant & Pellicer-Sánchez, Reference Tragant and Pellicer-Sánchez2019), audiovisual input (e.g., Peters & Webb, Reference Peters and Webb2018), subtitled audiovisual input (e.g., Peters, Reference Peters2019; Pujadas & Muñoz, Reference Pujadas and Muñoz2019), captioned audiovisual input (e.g., Montero Perez et al., Reference Montero Perez, Van Den Noortgate and Desmet2013; Peters, Reference Peters2019; Pujadas & Muñoz, Reference Pujadas and Muñoz2019), or glossed audiovisual input (e.g., Montero Perez et al., Reference Montero Perez, Peters and Desmet2018). What these types of input have in common is the combination of pictorial information (static or dynamic) and verbal input (spoken and/or written). In studies on multimodal input, learning gains are often explained in terms of the visual support (Peters, Reference Peters2019; Peters & Webb, Reference Peters and Webb2018; Rodgers, Reference Rodgers2018; Rodgers & Webb, Reference Rodgers and Webb2019; Wong & Samudra, Reference Wong and Samudra2019). However, most of these input types combine not two but three sources of input: (1) pictorial information, (2) written verbal information in captions or subtitles, or in written text, and (3) aural verbal input. In light of Baddeley’s (Reference Baddeley2007) dual-processing theory of working memory, it could be argued that language learners might experience cognitive overload when engaging with both pictorial and written information in addition to aural input. As a result of processing difficulties, language learners might divide their attention between the pictorial information and the written verbal information (Ayres & Sweller, Reference Ayres, Sweller and Mayer2014; Mayer & Moreno, Reference Mayer and Moreno1998). However, eye-tracking research has demonstrated that language learners are able to process both pictorial and written verbal information (Bisson et al., Reference Bisson, Van Heuven, Conklin and Tunney2012), on the condition that they are familiar with the script of the foreign language (Winke et al., Reference Winke, Sydorenko and Gass2013).

In addition to imagery, there are other advantages inherent in multimodal input and audiovisual input in particular. Learners need fewer words to understand TV programs compared to books. Webb and Rodgers (Reference Webb and Rodgers2009a, Reference Webb and Rodgers2009b) have put forward knowledge of the 3,000 most frequent word families and proper nouns to reach 95% coverage of the input. However, the lexical coverage figures for TV viewing have recently been found to be lower (Durbahn et al., Reference Durbahn, Rodgers and Peters2020), so the lexical demands are not as high as for reading (knowledge of the 4,000 most frequent word families for adequate comprehension and 8,000 word families for detailed comprehension; Laufer & Ravenhorst-Kalovski, Reference Laufer and Ravenhorst-Kalovski2010; Nation, Reference Nation2006). Rodgers and Webb (Reference Rodgers and Webb2011) also established that words are repeated more often in TV programs than in reading, especially in related TV programs, which is beneficial for vocabulary learning. Another advantage is the wide availability of audiovisual input using the Internet and streaming platforms. It can, thus, easily provide language learners with large amounts of authentic language input (Webb, Reference Webb, Nunan and Richards2015). Finally, language learners are motivated to watch L2 television, as has been well documented in surveys on language learners’ engagement with the L2 outside of the school (Lindgren & Muñoz, Reference Lindgren and Muñoz2013; Peters, Reference Peters2018; Peters et al., Reference Peters, Noreillie, Heylen, Bulté and Desmet2019).

LANGUAGE LEARNING FROM MULTIMODAL INPUT

Previous research into language learning from multimodal input has focused on three main areas: comprehension, vocabulary learning, and, to a lesser extent, grammar learning.

A consistent finding in this area is that audiovisual input is beneficial for comprehension, in particular when learners have access to captions (e.g., Gass et al., Reference Gass, Winke, Isbell and Ahn2019; Rodgers & Webb, Reference Rodgers and Webb2017; Winke et al., Reference Winke, Gass and Sydorenko2010; see also the meta-analysis by Montero Perez et al., Reference Montero Perez, Van Den Noortgate and Desmet2013). Captions assist comprehension by helping to break down speech into words and thus facilitating listening and reading comprehension. Crucially, a unique support offered to learners’ comprehension by multimodal input is imagery, as noted in the preceding text (e.g., Durbahn et al., Reference Durbahn, Rodgers and Peters2020; Jones & Plass, Reference Jones and Plass2002). Research into audiovisual input has shown that it can work as a compensatory mechanism especially for low-proficiency learners (e.g., Rodgers, Reference Rodgers2016).

The bulk of research into multimodal input has focused on vocabulary learning. A seminal study on the effect of TV viewing on vocabulary learning is Neuman and Koskinen’s Reference Neuman and Koskinen1992 study. They were among the first to stress the potential of audiovisual input for vocabulary learning. Other early examples of research into TV viewing include Vanderplank (Reference Vanderplank1988), Koolstra and Beentjes (Reference Koolstra and Beentjes1999), and the work by d’Ydewalle and colleagues (d’Ydewalle & Van de Poel, Reference d’Ydewalle and Van de Poel1999; Pavakanun & d’Ydewalle, Reference Pavakanun, d’Ydewalle, Engel, Bouwhuis, Bosser and d’Ydewalle1992). Although there were a considerable number of studies on TV viewing in the 1990s, audiovisual input was largely neglected by SLA researchers. It was not until 2009 that the field of SLA started to pay more attention to audiovisual input. Two key studies were the corpus studies by Webb and Rodgers (Reference Webb and Rodgers2009a, Reference Webb and Rodgers2009b), which showed the lexical demands of different types of audiovisual input. They argued that in addition to reading, audiovisual input may also be a valuable source of input for language learners. Since then, the field of SLA has witnessed a steady increase in the number of studies investigating vocabulary learning from audiovisual input (e.g., Sydorenko, Reference Sydorenko2010; Winke et al., Reference Winke, Gass and Sydorenko2010). A great deal of previous research into audiovisual input has focused on the efficacy of captions (= subtitles in the L2) on vocabulary learning (see Montero Perez et al.’s, Reference Montero Perez, Van Den Noortgate and Desmet2013, meta-analysis). Surprisingly, fewer studies have focused on noncaptioned and nonsubtitled audiovisual input (see Peters & Webb, Reference Peters and Webb2018, and Rodgers & Webb, Reference Rodgers and Webb2019, for two exceptions). Even though the majority of studies have been conducted with university students (e.g., Montero Perez et al., Reference Montero Perez, Peters, Clarebout and Desmet2014; Peters & Webb, Reference Peters and Webb2018; Rodgers & Webb, Reference Rodgers and Webb2019; Winke et al., Reference Winke, Gass and Sydorenko2010), there has been an increasing number of studies with learners in secondary schools (e.g., Peters, Reference Peters2019; Peters et al., Reference Peters, Heynen and Puimège2016; Pujadas & Muñoz, Reference Pujadas and Muñoz2019; Suárez & Gesa, Reference Suárez and Gesa2019), primary schools (e.g., Muñoz, Reference Muñoz2017), and even preschool children (e.g., Samudra et al., Reference Samudra, Wong and Neuman2019). Research has also moved from using short, educational clips to using full-length TV programs (e.g., Peters & Webb, Reference Peters and Webb2018) and even complete TV shows (e.g., Pujadas & Muñoz, Reference Pujadas and Muñoz2019; Rodgers & Webb, Reference Rodgers and Webb2019). Finally, in addition to studying the effectiveness of multimodal input for vocabulary learning, research has also started to study language learners’ processing of multimodal input (e.g., looking patterns of captions or pictures) by means of eye-tracking (Bisson et al., Reference Bisson, Van Heuven, Conklin and Tunney2012; Montero Perez et al., Reference Montero Perez, Peters and Desmet2015; Serrano & Pellicer-Sánchez, Reference Serrano and Pellicer-Sánchez2019; Winke et al., Reference Winke, Sydorenko and Gass2013; Wong & Samudra, Reference Wong and Samudra2019). Together, there seems to be robust evidence that language learners can indeed pick up unfamiliar words from multimodal input and that the provision of captions has the potential to increase the learning gains.

Research into grammar learning through multimodal input is very scarce. Early studies (d’Ydewalle & Van de Poel, Reference d’Ydewalle and Van de Poel1999; Van Lommel et al., Reference Van Lommel, Laenen and D’Ydewalle2006) found only small gains in young learners’ grammar learning under different subtitling conditions: reversed (L1 audio, L2 subtitles) and standard (L2 audio, L1 subtitles). In the study by Van Lommel and colleagues grammar learning gains were observed only when rules were presented in advance, which led these authors to conclude that, in contrast to vocabulary acquisition, grammar might be too complex to learn incidentally from a short video presentation. More recent studies involving captions and grammar in longer treatments have provided evidence of positive benefits for L2 grammar development in adults, especially when captions are textually enhanced (Lee & Révész, Reference Lee and Révész2018). However, results have not been similarly positive for all target structures, suggesting the influence of other factors such as the structure-specific saliency of a grammar token (Cintrón-Valentín et al., Reference Cintrón-Valentín, Garcia-Amaya and Ellis2019).

RATIONALE FOR AND OVERVIEW OF THE SPECIAL ISSUE

Research into the potential of multimodal input has been gaining traction, but the number of studies is still limited and mainly confined to vocabulary learning. Now that research into multimodal input is starting to broaden its focus to different aspects of learning as well as its research techniques, the present issue provides an up-to-date account of research in this area with a view to include innovative work and a range of approaches.

The special issue pursues new avenues in research into multimodal input by focusing on pronunciation, perception and segmentation skills, grammar, multiword units, and comprehension. In addition, it extends previous eye-tracking research (e.g., Winke et al., Reference Winke, Sydorenko and Gass2013) by investigating the effects of underresearched pedagogic interventions on learners’ processing of target items, target structures, and text. The studies nicely complement each other in their research methodologies and participant profiles. The special issue comprises six empirical studies and one concluding commentary.

The six empirical papers in the special issue explore:

• • Different types of input (TV viewing with and without L1 or L2 subtitles, reading-while-listening, reading, listening);

• • Different types of captioning (unenhanced, enhanced, no captioning);

• • Different components of language learning (single words, formulaic sequences, comprehension, grammar, pronunciation);

• • Different mediating learner- and item-related factors (e.g., working memory, prior vocabulary knowledge, frequency of occurrence);

• • Different learning conditions (incidental learning, intentional learning, experimental and classroom-based) and time conditions (short video clips, full-length TV programs, extensive viewing);

• • Participants of different ages (children, adolescents, adults); and

• • Different research tools (eye-tracking, productive and receptive vocabulary tests, comprehension tests).

Two studies in this special issue address vocabulary learning. They broaden the scope of existing research into multimodal input by comparing different types of input (written, spoken, audiovisual input in Feng & Webb) and by focusing on learning formulaic sequences (in Puimège & Peters).

Because several researchers (e.g., Webb, Reference Webb, Nunan and Richards2015) have recommended the use of audiovisual input for enlarging learners’ vocabulary, Feng and Webb’s study compared the relative effectiveness of three types of input (audiovisual, spoken, written) for vocabulary learning. It is the first study that compares reading, listening, and TV viewing in one experiment. To date, it is not clear whether audiovisual input is more effective than other types of input for language learning. To answer that question, Feng and Webb set up a quasi-experimental study, in which Chinese university students were assigned to one of four conditions: viewing condition, listening condition, reading condition, and control condition (= no input). The authors used a full-length TV program and tested EFL learners before and after the viewing on a set of target words. Participants in each experimental condition were exposed to the same content of the TV program, but in different ways depending on the condition to which they were assigned. The viewing group watched the TV program (audio + imagery); the reading group read the script of the TV program without having access to the audio; and the listening group listened to the audio of the TV program without having access to the visual support. Feng and Webb’s findings showed that all three groups learned vocabulary to the same extent. Regardless of the experimental group, there was incidental vocabulary learning from meaningful input. The authors conclude that in addition to extensive reading programs, there should “also be programs developed to encourage extensive listening and extensive viewing” (p. 520).

The study by Puimège and Peters explores the incidental learning of formulaic sequences from audiovisual input and factors affecting the learning of formulaic sequences. There is some evidence that formulaic sequences can be learned incidentally through reading (e.g., Pellicer-Sánchez, Reference Pellicer-Sánchez2017; Toomer & Elgort, Reference Toomer and Elgort2019) and reading-while-listening (Webb et al., Reference Webb, Newton and Chang2013). However, very little is known about learning formulaic sequences from audiovisual input, as the few studies into the effect of TV viewing so far have tended to focus on learning single words (Peters & Webb, Reference Peters and Webb2018). This is surprising given that corpus research has shown that the oral discourse typical of audiovisual input could provide sufficient exposure to formulaic sequences for incidental learning to occur (Lin, Reference Lin2014). Like Feng and Webb, Puimège and Peters used a full-length episode as input. By using a pretest-posttest, within-participant design, the authors found that Dutch-speaking English-as-a-foreign-language learners learned formulaic sequences both at the level of form recall and meaning recall after watching a 1-hour English-language documentary without subtitles. In line with previous research (e.g., Peters, Reference Peters2019; Peters & Webb, Reference Peters and Webb2018), they also found that learners’ prior vocabulary knowledge was positively related to learning (Matthew effect). Interestingly, the study shows the incremental nature of learning formulaic sequences, as learners’ preknowledge of the meaning of the formulaic sequences prior to the TV viewing was the strongest predictor of learning the form of formulaic sequences.

Two studies in this special issue focus on comprehension from multimodal input. Pujadas and Muñoz compared the differential effect of subtitles and captions on comprehension in a longitudinal, classroom-based study, while Pellicer-Sánchez et al. explore processing differences in reading-only and reading-while-listening.

The study by Pujadas and Muñoz contributes to the area by addressing the effects of captions and subtitles on TV viewing comprehension in a classroom intervention that extends over 8 months, which affords a unique longitudinal view with great ecological validity and pedagogical implications. Once a week four groups of adolescent beginner to low-intermediate learners of English viewed an episode of a sitcom series, two with captions and two with subtitles; one group in each language condition received explicit instruction of the target vocabulary prior to viewing the episode. The study also aimed at investigating how learners’ comprehension was affected by factors related to the learner, the episodes, and the testing instruments. The study found that, with learners at this age and proficiency level, viewing with subtitles leads to higher content comprehension than viewing with captions, and that having explicit instruction on target vocabulary had a small negative effect on comprehension (possibly because of the difficulty of splitting attention between the two demands). Other effects of on-screen text language were observed in the influence of vocabulary size only in the captions condition, and in differences in the way in which learners processed textually explicit information and inferential information, while the lexical coverage of the episodes was a significant predictor of comprehension independently of the language group.

Pellicer-Sánchez et al. make a unique contribution to research into multimodality by investigating young EFL learners’ processing of multimodal reading materials by means of eye-tracking. Compared to the number of studies focusing on university students, there is still little research into young language learners. The authors used a counterbalanced, within-participants design to explore processing differences in storybook reading with or without audio by analyzing learners’ looking patterns at the text and the pictures. All participants were exposed to both the reading-only and reading-while-listening condition. The authors found that learners spent proportionally more time on the pictures in the reading-while-listening condition than in the reading-only condition. However, it should be noted that this was not at the expense of text comprehension. Their study does not provide evidence for Mayer’s (Reference Mayer2009, Reference Mayer and Mayer2014) redundancy principle, which states that learning is negatively affected when verbal information is presented in both spoken and written form. On the contrary, their findings suggest that there was better integration of pictorial and verbal information when the storybook also had audio (reading-while-listening).

The final two studies both investigate the effect of captions, but each addresses a different aspect of learning, that is pronunciation development (Wisniewska & Mora) and grammar (Lee & Révész).

The study by Wisnieska and Mora is the first to examine the benefits of extended exposure to multimodal input through captioned and uncaptioned video on L2 pronunciation development. The scarce research in this area found benefits for L2 speech segmentation from captioned viewing (e.g., Charles & Trenkic, Reference Charles, Trenkic, Gambier, Caimi and Mariotti2015) and evidence that captions help retune L2 learners’ phonetic categories (Birulés-Muntané & Soto-Faraco, Reference Birulés-Muntané and Soto-Faraco2016; Mitterer & McQueen, Reference Mitterer and McQueen2009). Wisnieska and Mora extend previous research by examining processing skills and perceptual and productive accuracy in learners’ L2 phonology. Participants were randomly assigned to four experimental conditions involving two viewing modes (captioned or uncaptioned) and two task focus conditions (focus on phonetic form or focus on meaning). Benefits for L2 pronunciation were assessed in a pretest/posttest design after an 8-week exposure treatment. Unlike previous research, captioned video in this study was not superior to uncaptioned video in effecting gains in speech perception, which may reflect differences in testing. An intriguing result concerns differences in the benefits obtained for phonological accuracy under the two viewing modes and the two different attention-directing conditions. Captioned viewing led to pronunciation gains as long as there was no focus on phonetic form and learners’ attention was directed to meaning, whereas uncaptioned viewing led to pronunciation gains only when there was a focus on phonetic form. The authors interpret these findings as a case of cognitive overload that made it difficult to synchronize reading with online auditory speech processing. Like in Pujadas and Muñoz’s study (this issue), overloading may have resulted from asking learners to split their attention between competing tasks.

As noted in the preceding text, there is as yet little available evidence as to whether captioning may promote development in L2 grammatical knowledge. Lee and Révész’s pioneering study contributes to expand this area in multiple directions. Their study is novel in employing eye-tracking methodology to investigate whether attention allocation to target grammatical features is moderated by type of captioning and related to their development. The study followed a pretest-posttest-delayed posttest experimental design, and learners were assigned to an enhanced captions group, an unenhanced captions group, and a no captions group. The findings of the study show that captioning, with or without textual enhancement, facilitated the acquisition of grammatical knowledge, and that the enhanced captions group obtained greater gains. The authors suggest that in comparison with reading studies, where weaker effects are found (Lee & Huang, Reference Lee and Huang2008), textual enhancement and captioning might have increased the salience of the target features leading to a greater depth of processing. In addition, learners in the enhanced captioning condition paid more attention to the target grammatical construction and showed stronger association between attention and target construction development. This study has direct pedagogical implications illustrating how videos can be used effectively in meaningful tasks that are directly relevant for the language classroom (following a task-based language teaching approach) and exploring the effectiveness of directing learners’ attention to the target construction.

We are confident that this special issue will make a significant contribution to theories of language learning from multimodal input and will help us gain a deeper understanding of the mechanisms underlying language learning. The research questions addressed in this proposed special issue are also concerned with essential topics in the field, such as the effects of learner proficiency and individual differences (attention, working memory) or the effects of textual enhancement.

The research presented in this special issue demonstrates how multimodal materials can be used to increase and enrich learner exposure to language. Such research carries important educational implications. It speaks to language teachers indicating ways of enriching classroom input. It also speaks to language policy makers and society in general, increasing their awareness of the potential of audiovisual input for L2 learning outside the classroom, a learning context that is gaining prominence in the current globalized world.