2.1 The multimodal nature of audio-visual input

Video offers a combination of input modes including auditory (i.e., spoken input) and visual input. Visual input consists of dynamic imagery and can also include written on-screen text. Table 1 provides an overview of different types of on-screen text that have been studied in previous research. Because audio-visual input combines different input modes, it has also been characterized as a multimodal type of input. Multimodality has been defined as:

Table 1.

Types of on-screen text in studies published in SLA and CALL journals

Note. When multiple terms are available, the term that will be used in this article appears in bold.

Reversed subtitled input is not considered in this review because it does not expose learners to spoken L2 input.

Drawing on the importance of multimodal interaction and communication in our daily life, The Douglas Fir Group (2016) has defined L2 learning as multimodal and semiotic because language users and learners rely on a wide array of semiotic resources to construct meaning. In an everyday conversation, for instance, we use contextual information, visuals (e.g., face expressions), intonation, and so forth, to extract and build meaning. It has also been argued that multimodality has reached a ‘qualitatively new level’ (The Douglas Fir Group, 2016, p. 22) thanks to technological advances that have promoted the use of digital tools in daily life (e.g., online courses, videoconferencing).

The increasingly important role of multimodality may also impact how we approach texts. Serafini (Reference Serafini2012) proposed a reconceptualization of the reader as a ‘reader-viewer’ who uses specific skills to deal with multimodal texts. For instance, the reader-viewer is an interpreter who needs to construct meaning from the different aspects included in a multimodal text. According to Paivio (Reference Paivio1986, p. 53), human cognition:

Drawing on Paivio's dual coding-theory, Mayer (Reference Mayer2001) proposed the cognitive theory of multimedia learning. This theory posits that there are two separate channels for information processing that refer to presentation modes of information (i.e., verbal or nonverbal) or to sensory modalities (i.e., visual or auditory). According to the cognitive theory of multimedia learning, the simultaneous activation of these channels is expected to foster learning as indicated by the multimedia learning principle that holds that ‘people learn better from words and pictures than from words alone’ (Mayer, Reference Mayer2001, p. 63). Even though the cognitive theory of multimedia learning was initially developed and studied in the context of first language (L1) science learning, it has also been linked explicitly to L2 learning contexts (Mayer et al., Reference Mayer, Lee and Peebles2014). In addition, several of its principles, such as the spatial and temporal contiguity principle (see Section 3.1) and the redundancy principle (see Section 3.2), have been used in studies on L2 learning through watching audio-visual input.

2.2 Authenticity of language use in audio-visual input versus naturally occurring input

It has been asked whether language use in TV-programs and movies reflects real or authentic language use as compared with naturally occurring language use. This question has inspired corpus-driven research that is based on analyses of language use in specific corpora such as, for instance, The Sydney Corpus of Television Dialogue (Bednarek, Reference Bednarek2018), The TV Corpus, and The Movie Corpus (Davies, Reference Davies2021).

Overall, findings seem to indicate that television dialogue presents linguistic features of natural conversation even though it is often based on scripted language (e.g., language use in the TV sitcom Friends). For instance, it has been found that TV series are comparable to natural language use in terms of lexicogrammatical features (Bednarek, Reference Bednarek2018) and provide access to pragmatic information such as data on compliments and compliments responses (Rose, Reference Rose2001). TV dialogue also reflects colloquial and informal language use (i.e., spontaneous talk) and contains new expressions that reflect contemporary language use (Bednarek, Reference Bednarek2018).

The degree to which scripted language use presents natural language use may, however, depend on genre. Using Bibers’ Multidimensional analysis, Al-Surmi (Reference Al-Surmi2012) found that a sitcom series is closer to natural conversation than soap operas for some of the dimensions analyzed (e.g., expression of argumentation or persuasion), whereas soap operas were found to be closer to naturally occurring language use for narrative discourse (e.g., narrating events). However, sitcoms can also diverge from natural conversation in multiple ways. For example, language use in a corpus based on transcripts from the sitcom Friends contained fewer interruptions or overlaps (Quaglio, Reference Quaglio2010) than real language use. TV-series may also contain overrepresentations of certain features such as routine formulae or items such as ‘tell me what’ (Bednarek, Reference Bednarek2018). Further, it has also been argued that TV dialogue contains highly informal and non-standard elements of language use. In this respect, Werner (Reference Werner2020) argued that TV dialogue represents an interesting approach to illustrate grammaticality and informal spoken usage across varieties, which may contribute to learners’ language awareness. However, research in this particular area is scarce and does not yet reflect the diversity in types of audio-visual input that not only include TV programs but also YouTube videos or video-based social media (see Terantino, Reference Terantino2011, for a review on YouTube for foreign language learning).

Studies have also examined the authenticity of vocabulary use in TV series. For instance, Lin (Reference Lin2014) compared the distribution of formulaic sequences in a corpus of internet television to the spoken British National Corpus (BNC). She reported that the use of formulaic sequences in an internet television corpus reflects the frequency of formulaic sequences in everyday use. This highlights the potential of TV for learning formulaic language, as has also been found in experimental research (see Section 4). Sockett (Reference Sockett2012) investigated which four-word combinations occurred most frequently in a corpus of five television series. It was found that the 50 most frequently occurring four-word clusters correlated with real-life spoken English (as based on BNC data). Scheffler et al. (Reference Scheffler, Jones and Domińska2021) investigated vocabulary use in a corpus of an animation series (Peppa Pig) for pre-primary children. Results showed that 80% of the tokens in Peppa Pig stem from the 2,000 most frequent words in the BNC and that the 25 most frequently used words in the animation series are comparable to the 25 most frequent words in the spoken component of the BNC (i.e., high frequency pronouns, articles and prepositions). Together, these figures corroborate Vanderplank's definition of TV as ‘a very verbal medium’ (Vanderplank, Reference Vanderplank1993) by indicating that TV-series and programs reflect real language use and that exposure to these types of materials may hold great potential for L2 leaning.

2.3 Vocabulary demands of audio-visual input

Several corpus-based studies have sought to determine the vocabulary demands of different types of audio-visual input to identify the vocabulary size needed for adequate comprehension. Webb and Rodgers (Reference Webb and Rodgers2009a, Reference Webb and Rodgers2009b) analyzed the transcripts of, respectively, 318 movies and 88 TV programs such as news and TV series (e.g., sitcom, drama, sci-fi) in order to determine the vocabulary size needed for a 95% and 98% lexical coverage, that is, the amount of words in the input that a learner needs to know. These percentages had been defined as threshold percentages for adequate global and detailed comprehension in the context of written input (Hu & Nation, Reference Hu and Nation2000; Laufer & Ravenhorst-Kalovski, Reference Laufer and Ravenhorst-Kalovski2010).

Generally speaking, Webb and Rodgers found that knowledge of 3,000 word families (plus proper nouns and marginal words) is needed for a 95% coverage of movies (Webb & Rodgers, Reference Webb and Rodgers2009a) and TV programs (Webb & Rodgers, Reference Webb and Rodgers2009b). Yet, depending on the genre, figures of 2,000 to 4,000 word families (plus proper nouns and marginal words) have been put forward for 95% coverage. Studies on written input had found that knowledge of 4,000–5,000 most frequent word families is needed to achieve 95% coverage of written input (Laufer & Ravenhorst-Kalovski, Reference Laufer and Ravenhorst-Kalovski2010). A comparison of vocabulary demands of written and audio-visual input thus seems to indicate that vocabulary demands of audio-visual input are lower than for reading. In addition, there are different conditions that may lower the vocabulary load of video further such as, for instance, a narrow viewing approach.

Narrow viewing, which consists of watching related episodes of a TV-show (e.g., an entire season of a TV-series) or videos that are related to the same topic, can lower the vocabulary load for L2 learners. Results from a corpus-driven study (Rodgers & Webb, Reference Rodgers and Webb2011) that compared the transcripts of 288 television episodes revealed that related TV programs tend to contain fewer word families than unrelated TV-programs. Rodgers and Webb also found that lower-frequency word families tend to reoccur more often in related than in unrelated programs. Narrow viewing therefore has the potential to reduce the vocabulary load, which may subsequently contribute to L2 learners’ video comprehension. In addition, related programs may stimulate learners’ use of contextual and prior knowledge during the comprehension process because they become increasingly familiar with characters, plots, and settings.

Corpus-driven studies have also suggested that previewing activities can lower the vocabulary demands of audio-visual input. Webb (Reference Webb2010) found that pre-learning the ten most frequently occurring topic-related low-frequency word families can increase learners’ lexical coverage of a television programme from 0.70% to 3.91% and thus has the potential to improve learners’ comprehension of television programmes.

Finally, it has been argued that learners might achieve adequate comprehension of the input with coverage percentages lower than 95% when images are supportive of the meaning (Webb & Rodgers, Reference Webb and Rodgers2009a). Durbahn et al. (Reference Durbahn, Rodgers and Peters2020) measured L2 learners’ comprehension of an English documentary with a test that consisted of questions that were based on the audio, on the images, and on audio + images. They found that comprehension increases as learners’ lexical coverage increases. In addition, when learners can use images to answer the question, they rely less on their vocabulary knowledge than when they have to rely on audio alone. It should also be noted that their data did not reveal a threshold percentage for adequate comprehension. In sum, their findings suggest that imagery can be used to enhance comprehension and that lower levels of lexical coverage are needed in audio-visual input in which there is an important audio-imagery correlation.

However, not all videos and genres contain highly supportive imagery. Cross (Reference Cross2011) analyzed the audio-visual correspondence in news videotexts using the four-category coding system of Walma van der Molen (2001, as referenced in Cross, Reference Cross2011). Visuals were coded as talking head (i.e., no imagery beyond the person talking), direct (i.e., audio and visuals provide the same meaning), indirect (i.e., audio and visuals are not directly related), and divergent (i.e., audio and visuals are not related or there is a mismatch). His findings indicated that visuals in news videotexts are mainly talking heads, followed by indirect or divergent visuals. Images that were directly supportive of the audio were rare in the videos analyzed. Rodgers (Reference Rodgers2018) explored the potential of imagery for clarifying the meaning of unknown vocabulary in narrative TV genres (a fantasy/adventure TV series) and a documentary. It was found that both genres contained images that were supportive of the meaning of difficult words and that images in documentaries were more related to target vocabulary than narrative TV genres. The author links these findings to the spatial and temporal contiguity principle of the multimedia learning theory, which posits that ‘students learn better when corresponding words and pictures are presented near rather than far from each other’ (Mayer, Reference Mayer2001, p. 81) and that words and corresponding pictures should be ‘presented simultaneously rather than successively’ (Mayer, Reference Mayer2001, p. 96). In light of these principles, the images in documentaries were also found to co-occur (i.e., within a time frame of 5 seconds before or after) with the aural word form in 72% of the selected target words, which may promote information processing and learning (see Section 4.5). Together, these findings suggest that supportive imagery can lower the vocabulary demands of audio-visual input and that they may help learners in their meaning-making process, which makes the audio-image link an important criterium to consider when selecting audio-visual materials for L2 learning.

3.1 L2 learners’ video comprehension and the effects of captions

Already in 1988, Vanderplank argued that on-screen text such as captions (see Table 1) has the potential to ‘unlock’ input that would otherwise be too challenging (Vanderplank, Reference Vanderplank1988, p. 278). This was also found by Price (Reference Price1983) who reported on a large-scale experiment comparing the effects of watching video with and without captions on 500 university students’ content comprehension. Results revealed that captions yielded superior comprehension scores than video without captions and this was independent of language and educational background, social, and linguistic variables. While there is no detailed research report, it has inspired numerous researchers to investigate L2 learners’ comprehension of audio-visual input as well as ways to promote their comprehension of audio-visual input by means of captions.

So far, one meta-analysis (Montero Perez et al., Reference Montero Perez, Van Den Noortgate and Desmet2013), which was based on data of 15 studies published between 1989 and 2012, has investigated the effectiveness of captioned video as compared with uncaptioned video for L2 comprehension. A significantly large effect of captioning (g = 0.99) was found, which indicates that participants who were exposed to captions scored significantly better on comprehension tests than participants who were exposed to video without captions. More recent results are consistent with these findings (e.g., Majuddin et al., Reference Majuddin, Siyanova-Chanturia and Boers2021). The meta-analysis also revealed that the positive effects of captions were found irrespective of learners’ proficiency level.Footnote ³ However, the effect of captions was moderated by test type and the largest effects were found for captions when a receptive comprehension test was used (e.g., multiple choice) as compared with a productive test (e.g., recall test). The facilitative effect of captions on L2 comprehension has been explained by the fact that captions seem to assist L2 learners’ ‘attentional control’ (Gass et al., Reference Gass, Winke, Isbell and Ahn2019, p. 98) during input processing because they help learners to map the written and auditory trace. This may subsequently assist learners’ segmentation and decoding of the speech stream (see Section 5.2), which provides them with more time to process and interpret the meaning of the incoming information.

Using eye tracking, Kruger and Steyn (Reference Kruger and Steyn2014) explored the relationship between L2 learners’ comprehension of video and their captioning use. Participants watched six English educational videos in a captioning or no captioning group over a two-week period and took a comprehension test two weeks after the treatment. While no differences between the captioning and no captioning groups were found on the comprehension test, caption-reading behavior (i.e., learners’ fixations on the captions) was positively and significantly linked to performance on the comprehension test. The authors therefore argue that learners’ individual reading behavior should be considered in order to understand the global effect of captions on comprehension.

While numerous studies have revealed a positive effect of captions as compared with video without captions on L2 comprehension, the redundancy principle of the cognitive theory of multimedia learning would state that presenting the same information in different forms (e.g., audio and corresponding on-screen text) negatively affects learning. In addition, using on-screen text with video might lead to extraneous processing (Sweller, Reference Sweller and Mayer2005) in the visual channel that needs to process both the captions and the imagery, which could lead to a split-attention effect in the visual/pictorial channel. While these effects have been documented in numerous empirical studies dealing with science learning in learners’ L1 (see Adesope & Nesbit, Reference Adesope and Nesbit2012, for a meta-analysis on the redundancy effect), the aforementioned findings with L2 learners indicate that redundancy operates differently in L2 learning contexts. Mayer et al. (Reference Mayer, Fiorella and Stull2020) therefore propose ‘the subtitle principle’ that indicates that redundant information in the form of on-screen text may be helpful for comprehension in L2 learning contexts where learners have word identification problems.

3.2 Comprehension of (captioned) video under different viewing conditions

There are several viewing conditions that might influence L2 learners’ comprehension and in some cases the effectiveness of captioning. However, it should be noted that results remain provisional because of the limited number of studies in this area.

The number of viewings may influence L2 learners’ comprehension. Majuddin et al. (Reference Majuddin, Siyanova-Chanturia and Boers2021) found that viewing a video twice led to significantly higher comprehension scores than watching once, but this did not seem to interact with captioning effectiveness. Winke et al. (Reference Winke, Gass and Sydorenko2010) argued that when using repeated viewings, the order of viewing with captions (captions during the first viewing or during the second viewing) could play a role. They suggest that captions can be considered during the first viewing of the clip when the orthography of a learner's mother tongue is close to the script of the L2 (e.g., Spanish-speaking learners of English). Their data provide indications that captions might be more effective in a second viewing when the L1 and L2 represent greater orthographic distance (e.g., English-speaking learners of Arabic). However, it should be noted that the differences between conditions did not reach significance and that more research is needed to uncover whether learners’ L1 modulates the effectiveness of captioned video.

Mayer et al. (Reference Mayer, Lee and Peebles2014) investigated the role of captioning in relation to the pace of the video. They found that captioning did not lead to superior comprehension (compared with no captions) of a fast-paced narration. Fast-paced was defined as a video with 142 words per minute that is, however, comparable to naturally occurring speech containing approximately 150 words per minute. Drawing on these findings, they argue that the pace of the video should be slow enough in order to give learners time to process the on-screen text and avoid working memory overload (see, for instance, Frumuselu, Reference Frumuselu2018 and Liao et al., Reference Liao, Kruger and Doherty2020 on cognitive load and subtitles).

Narrow viewing, which consists in watching several related episodes (see section 2.3) may have an effect on captioning effectiveness. Using a multiple-session design, Rodgers and Webb (Reference Rodgers and Webb2017) investigated the effects of watching ten episodes of a TV-series (Chuck) in a captioned and uncaptioned condition on intermediate L2 learners’ comprehension. Participants watched ten 42-minute episodes (one per week) of the series and completed a comprehension test after each episode. It was found that all participants, irrespective of condition, significantly improved their comprehension scores from the first to the tenth episode, which indicates that regularly watching audio-visual input is beneficial for L2 learners’ listening comprehension. Findings also showed that learners with captions had significantly higher scores on the comprehension test than learners in the uncaptioned video for the first episode and for the episodes that were the most difficult ones. This seems to indicate that the large effect of captions in single-session treatments with short video exposure may be different in contexts with more sustained exposure to captioned video (i.e., longer clips over a longer period) and that input that is serial in nature may also facilitate comprehension irrespective of captioning availability because of learners’ growing knowledge of the series (i.e., characters, contexts, storylines).

3.3 Video comprehension and the length of captions

Several studies have investigated the question of whether captions should be full-length or only represent words that are important for the meaning of the excerpt, hence substantially reducing L2 learners’ reading effort while watching video. These studies draw on the work of Guillory (Reference Guillory1998) who found that beginning-level learners achieved similar comprehension scores in a full or keyword captioning group (representing only 14% of the full captions). She therefore argued that keywords are an interesting alternative to captions as they require less reading and have the potential to draw learners’ attention to specific ideas in the video.

Building on these findings, Montero Perez et al. (Reference Montero Perez, Peters and Desmet2014b) further explored the potential value of keyword captions. They found, however, that intermediate university-level learners who watched short videos with full captions outperformed the keyword group (who saw 14% of the words in the captions) and no captioning group on the global comprehension questions. Similar findings regarding the effects of keyword captions were reported by Teng (Reference Teng2019a) in a study with primary school learners. Additionally, Lee et al. (Reference Lee, Liu and Tseng2021) took into account learners’ self-reported caption reliance (i.e., how much they rely on captions while watching video) when comparing the effects of full, partial, and no captions on L2 learners’ comprehension. It was found that learners in a high captioning reliance group (see Section 6.2) performed best on a comprehension test after watching an 11-minute TED-talk with full captions (as compared with partial and no captions). However, learners in the low captioning reliance group were found to score best under a partial captioning condition. This raises questions about the role that individual differences and preferences might play in captioning effectiveness.

So far, results on the role of partial or keyword captions are inconclusive. Moreover, several potential caveats related to the use of keyword captions can be identified. First, the selection of words to show in the keyword caption mostly relies on manual selections. Yet, these selections, as well as the creation of keyword captions, are time-consuming – which may hinder the actual pedagogical use of such a tool. However, by using speech technology, Mirzaei et al. (Reference Mirzaei, Meshgi, Akita and Kawahara2017) were able to develop a partial (here, 30% of the caption text) and synchronized captioning tool. The words that appeared in the captions were automatically selected as a function of three parameters: speech rate, word frequency, and word specificity (i.e., technical vocabulary). It was found that participants in the partial and full captioning groups outperformed the control group, and that learners who had partial captions did not differ significantly from the full captioning group on a comprehension test, irrespective of their proficiency level. A second caveat is related to learners’ perceived usefulness of keyword captions. It has been found that learners perceived keywords as distracting and that they hampered their listening experience (Montero Perez et al., 2014). Finally, the percentage of words covered seems to vary across studies (e.g., from 14 to 30%) and may play an important role in explaining the mixed findings.

3.4 Video comprehension and the language(s) of on-screen text: Captions, subtitles, and bilingual subtitles

Whereas captions provide access to the written account of the L2 input, subtitles provide a translation of the audio in learners’ L1. Subtitles may therefore assist L2 learners’ comprehension of audio-visual input irrespective of L2 proficiency. The differential effects of subtitles and captions have been investigated in short clips (for early work on this topic, see Markham et al., Reference Markham, Peter and McCarthy2001) as well as full-length episodes of a TV series (e.g., Birulés-Muntané & Soto-Faraco, Reference Birulés-Muntané and Soto-Faraco2016; Vulchanova et al., Reference Vulchanova, Aurstad, Kvitnes and Eshuis2015). Findings point to an overall positive effect of subtitles on comprehension with learners who have access to subtitles outperforming learners in video only conditions (Vulchanova et al., Reference Vulchanova, Aurstad, Kvitnes and Eshuis2015) as well as in captioned and video only conditions (e.g., Bianchi & Ciabattoni, Reference Bianchi, Ciabattoni, Baldry, Pavesi and Taylor Torsello2008; Birulés-Muntané & Soto-Faraco, Reference Birulés-Muntané and Soto-Faraco2016) even though findings did not always reach significance levels (e.g. Liao et al., Reference Liao, Kruger and Doherty2020). Pujadas and Muñoz (Reference Pujadas and Muñoz2020) found that adolescent English as a Foreign Language (EFL) learners in a subtitling group also outperformed participants in a captioning group in an extensive viewing treatment in which they watched 24 episodes of a TV-series. In a study comparing different proficiency groups (i.e., beginning, intermediate, advanced learners), Lavaur and Bairstow (Reference Lavaur and Bairstow2011) found that beginning adolescent L2 learners benefitted most from subtitles as compared with captions and video only. No differences between the types of on-screen text were found for intermediate learners, whereas advanced learners benefitted most from video without on-screen text. Together, these findings suggest that subtitles can be considered effective for promoting L2 learners’ comprehension of audio-visual input and they seem to be especially relevant for making L2 audio-visual input available to beginning L2 learners who do not yet have sufficient reading speed in the L2 to deal with captions (Vanderplank, Reference Vanderplank2016a) or who do not yet have enough L2 knowledge to process captions efficiently (Muñoz, Reference Muñoz2017).

The role of bilingual subtitles (see Table 1 for a definition) has only recently attracted researchers’ attention. While this type of subtitling is less commonly found, it is often used in countries with multiple official languages (e.g., French and Dutch subtitles in movie theatres in Belgium) and they are increasingly being used in China (Liao et al., Reference Liao, Kruger and Doherty2020). The question whether bilingual subtitles are superior to other types of on-screen text for improving comprehension remains largely unanswered. Wang (Reference Wang2019), for instance, reported that short clips from an English TV series with bilingual subtitles were more effective than having the clips with captions for one class of freshman students. However, no differences were found for a different group of freshman students even though all participants were (higher) intermediate learners of English. Lwo and Chia-Tzu Lin (Reference Lwo and Chia-Tzu Lin2012) as well as Liao et al. (Reference Liao, Kruger and Doherty2020) did not find differences between a captioning, subtitling, bilingual subtitling, and video only group. One potential explanation for the lack of differences between these different conditions can be found in the eye-movement data reported in Liao et al. (Reference Liao, Kruger and Doherty2020). More specifically, they found that bilingual captions did not seem to significantly alter L2 learners’ viewing behavior, and that they tended to pick one language of the bilingual subtitle as main source of information. Participants thus spent similar amounts of time on the L1 text in the bilingual condition as on the text in the subtitle only group. Considering limitations in working memory capacity and the cognitive load theory, questions may be asked as to whether and when the simultaneous presentation of subtitles and captions is needed for L2 learners.

4.1 Vocabulary learning from video

During the last decade, there has been a surge of studies on vocabulary learning through watching L2 audio-visual input. Since this type of vocabulary learning takes place through contextual exposure, it can also be labelled contextual word learning (see Elgort et al., Reference Elgort, Brysbaert, Stevens and Van Assche2018 on contextual word learning from written input). Lin and Siyanova-Chanturia (Reference Lin, Siyanova-Chanturia, Nunan and Richards2015) argued that internet television may be an ideal approach to contextually learn new vocabulary because it may provide multimodal contextual cues. More particularly, this means that L2 learners can use verbal (spoken and/or written) as well as nonverbal (i.e., imagery) information to infer the meaning of new words.

To measure vocabulary learning that takes place through contextual exposure, previous studies have mainly relied on tests that tap into form-meaning mappings (see Table 2 for an overview of frequently used tests).

Table 2.

Frequently used types of vocabulary tests in research on L2 vocabulary learning from video (single words and multiword units)

Previous researchFootnote ⁴ has shown that video can be considered a source of vocabulary learning. Results indicate that watching a single TV program or a full-length episode has the potential to stimulate L2 learners’ incidental acquisitionFootnote ⁵ of single words. In Peters and Webb (Reference Peters and Webb2018), it was found that watching an English 1-hour documentary had a significant effect on L2 learners’ incidental vocabulary learning as measured by means of a meaning recognition and meaning recall test with approximately four words learned from pretest to posttest. Montero Perez (Reference Montero Perez2020) investigated a similar question but manipulated a French video to include 15 pseudowords, that is, the targets for incidental learning. Learning occurred at the level of form and meaning recognition with approximately six words learned on the form and three words learned on the meaning recognition tests. Other studies have expanded on these findings by showing that learners do not only acquire single words but also formulaic sequences from watching audio-visual input. Significant incidental learning of the form (Puimège & Peters, Reference Puimège and Peters2019a, Reference Puimège and Peters2019b) and meaning (Puimège & Peters, Reference Puimège and Peters2020) of formulaic sequences was found after watching an English documentary. Learning gains were influenced by several word-related factors including corpus frequency, collocate-node relationship (i.e., the combination of word classes or parts of speech within the formulaic sequence), semantic transparency (i.e., the degree to which the meaning of the formulaic sequence can be inferred from the constituent parts), as well as the association strength of the constituent components of the formulaic sequence.

Another question that has been asked is whether audio-visual input is comparable to written input in terms of the amount of contextual vocabulary learning that can take place (Webb, Reference Webb, Nunan and Richards2015). Arndt and Woore (Reference Arndt and Woore2018) compared incidental vocabulary learning of six pseudowords through YouTube videos and written blog posts. Vocabulary learning was measured by means of multiple tests targeting recall of orthographic form, meaning, and grammatical function as well as recognition of grammatical function and meaning. It was found that videos and blog posts were comparable in terms of learning gains but blog posts led to significantly higher gains on orthographic knowledge presumably because learners saw the written form of the pseudowords in the blog. Similarly, Feng and Webb (Reference Feng and Webb2020) compared incidental vocabulary learning through three modalities: written input, auditory input, and audio-visual input and found that all modalities led to vocabulary learning gains on a meaning recognition test. However, no differences were found between the groups, which seems to indicate that exposure to different input modes can lead to comparable incidental learning gains. As indicated by the authors, it is somewhat surprising that audio-visual input did not result in larger learning gains considering the potential value of imagery for word learning. They argue that this might be because their participants (i.e., EFL learners in China) were less familiar with the use of audio-visual input as learning materials.

4.2 Vocabulary learning from captioned video

Many studies have focused on whether vocabulary learning from video can be enhanced by means of captions. In a meta-analysis on captioned versus uncaptioned video (Montero Perez et al., Reference Montero Perez, Van Den Noortgate and Desmet2013), which was based on data from ten studies published before 2013, an overall large effect of captions on vocabulary acquisition (g = 0.87) was found, irrespective of the type of vocabulary test used and the proficiency level of the learners. This finding indicates that watching captioned video results in significantly larger vocabulary gains than uncaptioned video, which has been attributed to the fact that captions help learners to segment and decode the auditory input (see section 5.2). This may, in turn, facilitate L2 learners’ noticing of unknown words in the input. The superiority of captioned video as compared with uncaptioned video was further echoed in more recent findings for learning different aspects of word knowledge of single words such as form recognition (e.g., Chen et al., Reference Chen, Liu and Todd2018; Peters, Reference Peters2019; Sydorenko, Reference Sydorenko2010), meaning recognition (e.g., Teng, Reference Teng2019b), and meaning recall (e.g., Mohd Jelani & Boers, Reference Mohd Jelani and Boers2018; Teng, Reference Teng2019b), as well as primary school learners’ acquisition of verb-noun collocations (Teng, Reference Teng2019c).

While there is robust evidence to support the effectiveness of captions as compared with video only for L2 vocabulary learning, it has also been argued that the congruency between input and test modality might influence results of captioning effectiveness. More particularly, this means that the use of written vocabulary posttests in captioned video research may have favored participants who had access to captions. In order to address this issue, Sydorenko (Reference Sydorenko2010) presented half of the words in the recognition posttests in aural form and half in written form in a study with beginning learners of Russian. However, no input-test modality congruency effect was found: learners who had access to captions outperformed those in the uncaptioned group on the written and aural prompts of the meaning recognition test. Mohd Jelani and Boers (Reference Mohd Jelani and Boers2018), however, found that learners in the captioned group outperformed learners in the uncaptioned group on a meaning test for the written prompts but not for the aural ones. This seems to suggest that test modality should be taken into consideration in future research on this topic to obtain reliable measurements. Other studies have dealt with this risk by presenting written prompts and their corresponding spoken form simultaneously in vocabulary posttests (e.g., Peters, Reference Peters2019).

4.3 Vocabulary learning and the role of captions with textual enhancement, keyword captions, and glossed captions

While captions can stimulate learners to link spoken and written word forms while processing the audio-visual input, they do not overtly direct learners’ attention to new lexical units. In addition, while contextual clues and imagery can promote L2 learners’ acquisition of form-meaning mappings, they do not always provide explicit word meaning information. It has therefore been investigated how captions can be modified to promote noticing and form-meaning mappings.

Different techniques can be considered to foster learners’ noticing of vocabulary while watching captioned video. Studies have investigated the effects of making target words or structures in the captions visually salient through highlighting. Captions with highlighted target words were found to be more beneficial than captions without highlighted words for immediate word form recognition and form recall (Cintrón Valentín et al., Reference Cintrón Valentín, García-Amaya and Ellis2019) as well as meaning recognition (Montero Perez et al., Reference Montero Perez, Peters and Desmet2014b) and translation (Cintrón-Valentín & Garciá-Amaya, Reference Cintrón-Valentín and Garciá-Amaya2021) of single words. However, Cintrón-Valentín and García-Amaya (Reference Cintrón-Valentín and Garciá-Amaya2021) found that the advantage of highlighting was not maintained in the delayed posttests that were administered two weeks later. As argued by the authors, this might be explained by the fact that learners did not encounter the words in the two-week period preceding the delayed test while opportunities for retrieval practice are crucial to lead to retention.

Majuddin et al. (Reference Majuddin, Siyanova-Chanturia and Boers2021) further expand on the objectives of Cintrón-Valentín and Garciá-Amaya (Reference Cintrón-Valentín and Garciá-Amaya2021) by investigating the role of typographic enhancement realized through bolding and underlining multiword expressions in captioned video. They found that learners who had captions or captions with typographic enhancement picked up significantly more multiword expressions than a video only group but typographic enhancement did not lead to more gains than captioning without typographic enhancement. One feature that might explain the similar gains in the captioning groups is the length of the target structures used in the study. Since these could consist of up to five words, the effects of typographic enhancement may have been minimized.

Studies have also investigated the role of visual salience operationalized through the presence of keyword captions. However, the impact of keyword captions on vocabulary learning remains unclear. While keywords were found to be as effective as captions and captions with highlighted keywords for adult intermediate learners’ vocabulary learning (Montero Perez et al., Reference Montero Perez, Peters, Clarebout and Desmet2014a), captions seemed to be more effective for the learning of collocations than keyword captions in primary school EFL learners (Teng, Reference Teng2019c).

In order to promote learners’ form-meaning mappings, glossed captions can be considered. Glosses in captions can be accessed by clicking on a word in the caption in order to see its meaning such as the L1 translation (see Boers, Reference Boers2021 for a review on glosses for vocabulary learning). While research on the use of glossing in audio-visual input is scarce, it has been found that glosses that provide access to an L1 translation can result in significantly more form-meaning mappings than full or keyword captioning (Hsieh, Reference Hsieh2020; Montero Perez et al., Reference Montero Perez, Peters and Desmet2018; Teng, Reference Teng2020). The advantage of glossed captions is that learners have immediate access to a translation and do not need to rely on the availability of contextual cues or imagery to infer word meaning from video. In addition, positive relationships have been found between gloss consultation and vocabulary posttest scores after watching short clips (Montero Perez et al., Reference Montero Perez, Peters and Desmet2018) and multiple full-length episodes of a TV-series (Fievez et al., Reference Fievez, Montero Perez, Cornillie and Desmet2021), which further strengthen the potential of glossing form the installation of form-meaning mappings.

In sum, different techniques to make vocabulary more salient in the captions (i.e., highlighting, bolding, keywords) have been investigated but they have yielded mixed results. Results regarding the positive effect of glossed captions seem to be more clear-cut despite the limited number of studies in this area.

4.4 Vocabulary learning from subtitled video

Early work on the role of subtitled video for vocabulary learning has been conducted with young L2 learners (e.g., Koolstra & Beentjes, Reference Koolstra and Beentjes1999; d'Ydewalle & Van De Poel, Reference d'Ydewalle and Van de Poel1999) and revealed that children who had subtitled audio-visual input picked up more words than learners who did not have subtitles.

More recently, the effects of subtitled video have been compared with captioned video. Results revealed that captions lead to higher scores than subtitles on form recognition, form recall, and meaning recall tests of (low-) intermediate learners (e.g., Peters, Reference Peters2019; Peters et al., Reference Peters, Heynen and Puimège2016) after a one-off intervention. Captions also yielded better results than subtitles for promoting L2 learning of colloquial speech after watching 325 minutes of a TV series (Frumuselu et al., Reference Frumuselu, De Maeyer, Donche and Gutiérrez-Colon Plana2015). Other studies comparing subtitles and captions did not yield significant differences on beginning (Bisson et al., Reference Bisson, van Heuven, Conklin and Tunney2014) or low to intermediate L2 learners’ meaning and form recall (Pujadas & Muñoz, Reference Pujadas and Muñoz2019) or meaning recognition (Birulés-Muntané & Soto-Faraco, Reference Birulés-Muntané and Soto-Faraco2016) scores even after larger amounts of exposure (i.e., >110 min) to TV-programs (Fievez et al., Reference Fievez, Montero Perez, Cornillie and Desmet2020; Muñoz et al., Reference Muñoz, Pujadas and Pattemore2021; Pujadas & Muñoz, Reference Pujadas and Muñoz2019; Frumuselu, Reference Frumuselu, Herrero and Vanderschelden2019).

While differences in terms of design and amount of exposure make it hard to compare findings across studies, results do not seem to highlight specific advantages of subtitles as compared with captions for L2 word learning. One possible explanation that may account for this finding is that subtitles do not facilitate speech stream segmentation and sound-script mapping, which may decrease learners’ chances of noticing L2 forms and establishing form-meaning links.

4.5 Vocabulary learning from (captioned and subtitled) video and factors that influence learning

Even though there is a fair amount of research on vocabulary learning from audio-visual input, it is hard to determine and compare learning gains across studies because of the wide variation in terms of methodology and treatments. An analysis of the treatments used shows that they can be categorized in three groups based on the length of the treatment and the number of treatment sessions: (1) interventions based on short clips resulting in exposure of 9 to 20 minutes in single-session designs (e.g., Peters et al., Reference Peters, Heynen and Puimège2016; Sydorenko, Reference Sydorenko2010); (2) interventions that have used longer clips (i.e., full-length episodes of TV programs) between 20 and 60 minutes in single-session designs (e.g., Birulés-Muntané & Soto-Faraco, Reference Birulés-Muntané and Soto-Faraco2016; Peters & Webb, Reference Peters and Webb2018); and (3) interventions that are based on multiple full-length episodes (e.g., Rodgers & Webb, Reference Rodgers and Webb2020) in a multiple-session design (i.e., between 7 and 24 sessions). Learning gains found from pretest to posttest range from approximately 1 to 14 words learnedFootnote ⁶ on a meaning recall or recognition test. However, these gains do not only depend on the presence of on-screen text, the duration of the treatment, and the number of sessions, but also on several other factors including word-related and learner-related variables.

Several properties of the target words – the words that are included in the pre- and posttests – may influence learning gains from watching (captioned or subtitled) audio-visual input. One such factor is cognateness, which has been defined as words from different languages that present an important overlap in form and meaning (e.g., the English word monetary and the Dutch word monetair in Peters & Webb, Reference Peters and Webb2018). It has been found that cognateness has a facilitative effect on incidental vocabulary learning through watching (captioned or subtitled) audio-visual input (e.g., Fievez et al., Reference Fievez, Montero Perez, Cornillie and Desmet2020). In Peters and Webb (Reference Peters and Webb2018), for instance, the odds of a correct response on the meaning recall test were eight times higher when the target word was a cognate. These findings seem to indicate that there is a cognate facilitation effect in incidental vocabulary learning from video, which is consonant with other studies in the field of L2 vocabulary learning and processing (see Peters, Reference Peters and Webb2020, for a comprehensive review).

Frequency of occurrence, that is, the number of instances of a word in the input, can have a facilitative effect on vocabulary learning from video as well. Repeated encounters have been found to positively influence the learning of single words through (captioned) audio-visual input in studies using a single viewing session (e.g., Peters & Webb, Reference Peters and Webb2018) as well as after watching several videos (e.g., Rodgers & Webb, Reference Rodgers and Webb2020). The positive effect of frequency is in line with studies on the role of frequency in written input but it remains unclear how many encounters are needed through audio-visual input in order to result in considerable learning. In studies using a multiple-session design (i.e., multiple viewing sessions), stronger effects of frequency on word learning have been reported when words appear repeatedly in a single episode – that is, through massed repetition – than when they appear spaced across episodes (e.g., Fievez et al., Reference Fievez, Montero Perez, Cornillie and Desmet2021; Rodgers & Webb, Reference Rodgers and Webb2020). This is in line with findings from a meta-analysis (Uchihara et al., Reference Uchihara, Webb and Yanagisawa2019) that revealed that massed conditions yielded larger effects than spaced conditions in incidental vocabulary learning research.Footnote ⁷

Words that are accompanied by imagery (i.e., visual support provided by the images used in the video) may lead to higher scores on vocabulary posttests. Sydorenko's results (Reference Sydorenko2010), for instance, showed that beginning learners had higher scores on a translation test for words that were depicted in the images than for words without visual support after watching short excerpts from a Russian comedy series. She therefore concluded that ‘the meanings of some new words can be learned from very difficult authentic videos when the language is well-supported by visual images’ (2010, p. 61). The supportive role of imagery for learning the meaning of unknown words was further confirmed by Peters (Reference Peters2019) who found that learners were three times more likely to be correct on a form recognition and meaning recall test when the meaning of target words was clarified in the images that occurred in a timeframe of five seconds before or after the use of the target in the auditory trace of the documentary (i.e., the spatial and temporal contiguity principle as presented in section 3.1).

Finally, word learning from watching audio-visual input is also affected by learner-related variables. There is a growing body of literature that has identified learners’ prior vocabulary knowledge as an important variable in determining the amount of incidental learning that takes place while watching (captioned and subtitled) audio-visual input. More particularly, higher learning gains have been reported for learners with larger vocabulary sizes (e.g., Montero Perez, Reference Montero Perez2020).Footnote ⁸ This is often referred to as the Matthew effect or the ‘rich get richer principle’ as was already reported in early work on this topic (Baltova, Reference Baltova1999) in which it was also found that learners with a better oral fluency learned more vocabulary (Neuman & Koskinen, Reference Neuman and Koskinen1992). This has been explained by the fact that learners who know more words tend to have a better lexical coverage of the input, which may help learners to segment the speech stream and to notice unknown words in the input. Similarly, learners with larger scores on general proficiency tests have also been found to obtain larger vocabulary learning gains after watching captioned or subtitled video than learners with lower scores (Muñoz et al., Reference Muñoz, Pujadas and Pattemore2021).

In order to deal with the moderating role of word-related and learner-related factors, studies have recently shifted from analyses of (co)variance towards logistic regression and mixed effects models. The advantage of these types of analyses is that they are based on items and therefore allow to include different input-related variables such as the frequency of occurrence of words in the input, their visual representation, cognateness, as well as learner-related variables (e.g., L2 learner’ proficiency level) in one model. This is valuable when target items have different properties as is typically the case in studies on audio-visual input because of the difficulty of manipulating target items in the input. However, it remains crucial to consider word-related factors when selecting target words.

5.1 Learning grammar and syntax through watching video with different types of on-screen text

The effects of audio-visual input and on-screen text on the learning of grammar have remained largely unexplored. Early work (e.g., d'Ydewalle & Van De Poel, Reference d'Ydewalle and Van de Poel1999) on this topic found that children (8–12 years old) did not seem to show significant learning of morphology or syntax in the L2 (French or Danish) after watching a 10-minute video with subtitles or reversed subtitles (see Table 1 for a definition). Similarly, Van Lommel et al. (Reference Van Lommel, Laenen and d'Ydewalle2006) investigated the acquisition of Esperanto grammar rules after a 16-minute subtitled video. Results indicated that watching the video without explicit instruction about grammar rules did not result in grammar uptake with scores around chance level. This finding is not surprising and might be explained by the nature of grammar learning that consists of the acquisition of grammatical regularities. To abstract these regularities incidentally from input, large amounts of exposure including repeated encounters with target constructions will be needed for learners to apply these rules to new utterances. This distinguishes grammar learning from vocabulary learning, which consists of learning items rather than regularities.

Research into the effects of more substantial exposure to L2 input on the learning of grammar and syntax is scarce. One exception is the study by Ghia (Reference Ghia2012), who investigated whether watching five subtitled films could stimulate L2 learners’ acquisition of syntax (i.e., the ordering of main constituents within phrases and clauses such as clefts, interrogative clauses, and complex phrases). More specifically, she studied the concept of ‘translational salience’, which refers to the relationship between the L2 audio and the translation of the audio in the subtitles. For instance, subtitles can present reductions or substitutions or literal transfer between L1 and L2. Subtitles can also include ‘discrepancies in subtitle translation’ (p. 90). Results indicated that the viewing activity resulted in initial learning of L2 syntax irrespective of the translation strategy used.

Pattemore and Muñoz (Reference Pattemore and Muñoz2020) investigated whether watching ten episodes of a TV-series (totaling 227 min) could stimulate L2 learners’ acquisition of grammar. Participants were undergraduate EFL learners who were assigned to a captioned video or uncaptioned video condition. The authors found significant increases from pretest to posttest on the grammatical constructions test and students in the captioned group outperformed learners in the uncaptioned group. A significant interaction effect between working memory and the presence of captioning was also reported, which revealed that learners who did not have captions seemed to rely more on their working memory than learners who had captions, irrespective of their proficiency level. In addition, working memory – that is, the system that temporarily stores and processes information during higher order task performance – was linked to grammar learning gains in the uncaptioned group with better learning of grammatical constructions found in learners with greater working memory capacity. This is in line with Gass et al. (Reference Gass, Winke, Isbell and Ahn2019, p. 84), who argued that captions ‘may neutralize some of working memory's limiting effects on language learning’. In a follow-up study using similar input conditions (Muñoz et al., Reference Muñoz, Pujadas and Pattemore2021), it was found that frequency of occurrence of target grammatical constructions correlated with grammar learning gains of upper-intermediate university L2 learners, especially for learners who did not have access to captions.

Drawing on the importance of noticing for L2 learning, captions with typographical enhancement have been examined. Lee and Révész (Reference Lee and Révész2020) found that captions with highlighted structures helped advanced EFL learners to outperform a captioning and a control group, even with treatment tasks that were based on 24 short clips of 20 to 50 seconds each. The use of captioned video with enhancement thus seems to encourage learners to reflect more about the target structures, hence resulting in learning gains. Similarly, Cintrón Valentín et al. (Reference Cintrón Valentín, García-Amaya and Ellis2019) investigated the effects of captions with/without highlighted target structures on L2 grammar learning. Crucially, participants watched multimodal input that was specifically created for this study and that consisted of animated videos in which the target items were carefully controlled for frequency of occurrence as well as for their placement in the video. They reported positive effects of textually enhanced captions on the learning of some of the selected target grammatical structures. The authors also argued that some target grammatical structures have structure-specific characteristics which may have mediated the effects of captioning and highlighting. This finding was further echoed in Cintrón-Valentín and Garciá-Amaya (Reference Cintrón-Valentín and Garciá-Amaya2021).

5.2 Effects of (captioned and subtitled) audio-visual input on L2 listening and pronunciation

Being able to process the speech stream and automatically recognize word boundaries of L2 speech – that is, learners’ bottom-up listening processes (Rost, Reference Rost2002) – is essential for successful L2 listening comprehension. To promote L2 learners’ bottom-up processes, studies have, for instance, used repeated or intensive listening activities in which learners process a particular input repeatedly and segment per segment (e.g., Matthews et al., Reference Matthews, Cheng and O'Toole2014 on audio-based listening). Another approach consists of providing L2 learners with captioned video. In past research, captioned video had, however, been criticized because captions were expected to hinder listening (Diao et al., Reference Diao, Chandler and Sweller2007, on audio-based listening) as well as the development of strategies needed to deal with authentic listening situations (Pujolà, Reference Pujolà2002; Vandergrift, Reference Vandergrift2004). However, there are now first indications that captioned video stimulates speech perception (see Bird & Williams, Reference Bird and Williams2002 for early work on this topic) in such a way that its effects are generalizable beyond the input used in the treatments. This was the case in Charles and Trenkic (Reference Charles, Trenkic, Gambier, Caimi and Mariotti2015), who report on a small-scale study investigating advanced L2 learners’ speech segmentation ability after watching two 30-minute excerpts from two English documentaries in one of the three conditions (captioned video, sound only, or text only). It was found that participants in the captioned group showed most improvement on the speech segmentation test, which consisted of known sentences, but also unknown ones.

Other studies have examined the value of captioned video as compared with subtitled video for L2 speech decoding. Mitterer and McQueen (Reference Mitterer and McQueen2009) found that advanced users of English could adapt to an unfamiliar accent after only 25 minutes of exposure to captioned video. In addition, captions were also found to lead to enhanced perceptual adaptation for both new and previously encountered words. Subtitles, however, were only helpful for known utterances and seemed to ‘create lexical interference’ (Mitterer & McQueen, Reference Mitterer and McQueen2009, p. 1), which hindered speech adaptation for new items. Furthermore, Birulés-Muntané and Soto-Faraco (Reference Birulés-Muntané and Soto-Faraco2016) reported that watching a one-hour episode of an English TV drama with captions led to significantly more improvement of higher-intermediate English as a Second Language (ESL) learners’ scores on a fill-gap listening test (i.e., learners had to fill in 24 words in a text while listening to a fragment) than watching video with subtitles or video only. However, they also pointed to a potential trade-off between subtitles and captions. Whereas the development of L2 speech perception was better with captions, scores on the plot comprehension test were highest with subtitles. Wisniewska and Mora (Reference Wisniewska and Mora2020) found that an eight-week intervention program in which advanced learners watched 16 episodes of the British TV-series Luther with or without captions influenced L2 learners’ speech processing skills. Importantly, the intervention program did not only stimulate learners’ speech segmentation but also enabled them to process speech in a faster and more efficient way, both of which are crucial in the development of listening fluency (Rost, Reference Rost, Muller, Adamson, Shigeo Brown and Herder2014). Surprisingly, no differences between the captioned and video-only group were found. This might be owing to the advanced proficiency level of the participants or to the treatment, which was substantially longer than the treatment used in the abovementioned articles. Finally, Wisniewska and Mora found that captioned video led to pronunciation gains, as previously suggested by Mora and Cerviño-Povedano (Reference Mora, Cerviño-Povedano, Herrero and Vanderschelden2019). However, gains were also found in the uncaptioned video condition when learners’ attention was directed to phonetic form.

6.1 L2 learners’ perceptions towards the use of (captioned and subtitled) audio-visual input for L2 learning

Few studies have focused explicitly on perceptions and attitudes towards using video for L2 learning. There are, however, indications that learners find audio-visual input motivating and consider it a useful resource for L2 learning (e.g., Sockett & Kusyk, Reference Sockett, Kusyk, Cadierno and Eskildsen2015; Wang, Reference Wang2012). Regarding the use of on-screen text, it has been found that captioned video may result in more relaxed viewing (Pujadas & Muñoz, Reference Pujadas and Muñoz2019) and may increase learners’ self-confidence as language learners (Teng, Reference Teng2021). In addition, captions do not seem to distract learners or hamper their enjoyment (Kruger et al., Reference Kruger, Doherty and Soto-Sanfiel2017). In one of the few studies based on qualitative data, Vanderplank (Reference Vanderplank2019) investigated L2 learners’ use of captioned video in an informal context. Participants watched video with captions over a period of six to 12 weeks. Importantly, they were in control of the viewing material. An analysis of viewing diaries revealed changes in attitudes and viewing behavior, resulting in three different usage groups: minimal users whose focus was on enjoying films, maximal users who used films for L2 learning, and evolving users who reported using films increasingly as a tool to learn languages. Vanderplank also argued that learners might need advice or guidance in order to maximally benefit from L2 audio-visual input.

In a large-scale study, Mariotti (Reference Mariotti, Gambier, Caimi and Mariotti2015) focused on L2 learners’ use of audio-visual input with on-screen text. Participants were 1,374 secondary school and university students enrolled in formal language learning and 555 adult learners who watched video with on-screen text in an informal learning context. Participants who watched subtitled videoFootnote ⁹ in a formal context thought that it had a positive effect on their listening and speaking skills. In addition, 70% of the students indicated that they watched more subtitled video in their spare time. However, students also indicated several problems such as the speed of the subtitles, which was often perceived as too fast. Of the participants who used video with on-screen text in an informal context, 60% found that it was time well spent and thought it helped them improve their listening skills. Most of the participants seemed to prefer subtitles (rather than captions or reversed subtitles) when watching for pleasure. In addition, learners who felt more confident also watched more subtitled video and reported to be more active in conversations. Finally, they also observed that learners who were very motivated to learn the L2 preferred captions instead of subtitles.

6.2 L2 learners’ self-reported use of input modes in (captioned) video

Findings from self-reported data on L2 learners’ use of input modes (i.e., audio, images, on-screen text) in captioned video (Sydorenko, Reference Sydorenko2010) indicated that beginning L2 learners primarily focused on the on-screen text, followed by the images and the audio. However, learners found the visuals more useful than the captions and the audio. In order to assess how much learners rely on captions (i.e., textual information), Leveridge and Yang (Reference Leveridge and Yang2013) developed a 40-item multiple-choice test including questions on sections where the captions match the audio (congruent items) and questions that focus on excerpts in which captions and audio do not match (incongruent items). The score gives an indication of learners’ caption reliance and it was found that learners’ score correlates with their perceived captioning reliance (Leveridge & Yang, Reference Leveridge and Yang2014). However, this information is based on self-reported data and it does not include information on learners’ use of images.

Drawing on the concept of multimodality (see Section 2.1), studies have recently tried to tap into learners’ use of a wide range of semiotic resources when watching audio-visual input. Compared with previous research in which a distinction was made between images, on-screen text, and audio, Ruck (Reference Ruck2020) investigated L2 learners’ self-reported foci of attention on 12 possible sources of input in the videos (i.e., an excerpt from a film and from a documentary) such as vocabulary, visuals, (non linguistic) sounds, cultural products (e.g., books, tools) or behaviors. She found that learners’ self-reported use of these sources, as measured by means of a questionnaire, differed in both genres and that learners paid, for instance, more attention to linguistic features in the documentary than in the film clip. Abrams (Reference Abrams2016) included a TV-series in technology-mediated task cycles for beginning L2 learners of German. She investigated the potential contribution of tasks to learners’ semiotic awareness, that is, their awareness to use different aspects of multimodal input for meaning extraction. She found that the use of captioned audio-visual input contributed to learners’ awareness of the connections between different semiotic resources such as, for instance, the link between actions and body language depicted in the input.

6.3 L2 learners’ use and processing of captions and subtitles: Findings from eye-movement studies

An important addition to the literature on L2 audio-visual input since Vanderplank's (Reference Vanderplank2010) article is the use of online data such as eye tracking in order to understand learners’ use and processing of input modalities (see Doherty & Kruger, Reference Doherty, Kruger, Dwyer, Perkins, Redmond and Sita2018 and Kruger et al., Reference Kruger, Szarkowska and Krejtz2015 for reviews on eye tracking in subtitling research).Footnote ¹⁰ Eye-movement technology allows for relatively unobtrusive data collection and makes it possible to register eye movements while participants engage in a natural viewing activity (see Godfroid, Reference Godfroid2020 and Conklin et al., Reference Conklin, Pellicer-Sánchez and Carrol2018 for methodological discussions on eye-tracking and multimodal input). It should be noted that eye-tracking studies with audio-visual input are relatively scarce and have focused on two main areas of interest, that is, learners’ time spent on the input modalities (i.e., on-screen text and images) and learners’ time spent on specific words or structures in the on-screen text. So far, however, the question of how the dynamic nature of the input (i.e., moving images with moving text that is synchronized with audio) impacts learners’ reading processes remains largely unanswered.Footnote ¹¹

With regard to the amount of time spent on the captioning or subtitling area, Bisson et al. (Reference Bisson, van Heuven, Conklin and Tunney2014) reported reading percentages of 43% for the captions and 51% for subtitles. Even though learners were not familiar with the language used in the captions (i.e., Dutch), findings reveal that they spent a significant amount of time on the on-screen text. The amount of time spent on the captions in Winke et al. (Reference Winke, Sydorenko and Gass2013) was slightly higher than in Bisson et al. (Reference Bisson, van Heuven, Conklin and Tunney2014), with 66% for familiar content and 69% for unfamiliar content. Differences in script may, however, affect eye-movements. For instance, L2 learners of Arabic spent more time reading the captions than L2 learners of Spanish and Russian. Fixation times of L2 learners of Chinese presented a lot of variation and they varied in function of content familiarity (74% for unfamiliar vs. 62% for familiar content). In a study with different age and proficiency groups, Muñoz (Reference Muñoz2017) reported that children (who were mainly beginning EFL learners) tended to skip subtitles less and had a longer total fixation duration on the subtitle area than older learners. More fixations were found on captions than on subtitles in children's fixation data, which may indicate that captions are more challenging to process for this age and proficiency group. Older and higher-level learners showed a more regular reading pattern of the captions but tended to skip subtitles more often. Finally, working memory has also been found to play a role in L2 English learners’ time spent on captions (Gass et al., Reference Gass, Winke, Isbell and Ahn2019). Findings indicated, for instance, that learners with a higher working memory spent less time rereading the captions during a second viewing.

Another question that has been addressed is whether learners are able to simultaneously deal with on-screen text and imagery, which are both visual input sources. Fixation times from eye-movement analyses have shown that learners are able to process the images even when subtitles or captions are shown and read (e.g., Bisson et al., Reference Bisson, van Heuven, Conklin and Tunney2014). This is in line with earlier work that had shown that learners are able to shift their attention between images and subtitles effortlessly and automatically (d'Ydewalle & Gielen, Reference d'Ydewalle, Gielen and Rayner1992) while watching a subtitled program with two-line subtitles (d'Ydewalle & De Bruycker, Reference d'Ydewalle and De Bruycker2007). Following up on d'Ydewalle and De Bruycker (Reference d'Ydewalle and De Bruycker2007), Perego et al. (Reference Perego, Del Missier, Porta and Mosconi2010) found similar performance on a subtitle recognition test (i.e., multiple choice questions on the exact wording in the subtitle) and scene recognition task (i.e., ‘Did you see the scene in the video?’) and reported that scene recognition was still successful even if learners spent less than 40% of their fixations on the visuals. According to these data, L2 learners can process and integrate information from multiple competing sources of information. These findings also point to the cognitive effectiveness of subtitle and image processing and suggest that there does not seem to be a trade-off between the processing of images and subtitles.

Eye-tracking has also been used to quantify learners’ time spent on specific words or structures in on-screen text as a function of attention-drawing techniques. Regarding the effect of textual enhancement through highlighting, it was found that highlighting words in captions (Lee & Révész, Reference Lee and Révész2020) was successful in directing learners’ attention to the target grammatical constructions with significantly more rereading times and visits to highlighted target structures. Higher rereading times were also found by Ghia (Reference Ghia2012) who used translational salience (i.e., including a contrast between the L2 structure and the L1 translation in the L1 subtitle) as attention-drawing technique. Eye-movement analyses in Montero Perez et al. (Reference Montero Perez, Peters and Desmet2015) revealed an interplay between two attention-drawing techniques: visual salience (full vs. keyword captions) and test announcement (i.e., informing vs. not informing learners about upcoming vocabulary tests).

Finally, it has been asked whether learners’ fixations on target items are related to learning gains. In order to do so, the relationship between eye-movement data (i.e., online measures) and offline measures (i.e., learners’ scores on posttests) have been considered (see Godfroid et al., Reference Godfroid, Boers and Housen2013 on incidental vocabulary learning while reading). For instance, Lee and Révész (Reference Lee and Révész2020) found that attention to the target structures in the captions with highlights was linked to greater performance on a written and oral productive test but only for one of the target grammatical structures (i.e., present perfect). Montero Perez et al. (Reference Montero Perez, Peters and Desmet2015) found that eye-movement data of learners in the captioning incidental learning group significantly correlated with learning. For instance, learners’ first pass time (i.e., the amount of time spent on a word before the eyes leave that word) on a given word correlated positively with L2 learners’ form recognition of that word, indicating that longer first pass time increases the likelihood of correctly recognizing a word on the posttest.

6.4 L2 learners’ exposure to audio-visual input outside of the L2 classroom and its effect on L2 learning

Recently, studies on informal language learning or language learning beyond the classroom (see Reinders & Benson, Reference Reinders and Benson2017 on dimensions and definitions of these concepts) have identified audio-visual input as an important source of L2 input in out-of-school contexts. This is not surprising since audio-visual input is ubiquitous and can be easily encountered in daily life. It has, for instance, been shown that watching audio-visual input (in the L1 or the L2) is one of the most common activities in everyday life (European Commission, 2019) with 81% of the people who filled out the survey watching TV (almost) every day.

It has been reported that L2 learners of English frequently engage with audio-visual input in the target language in out-of-school contexts (e.g., Lindgren & Muñoz, Reference Lindgren and Muñoz2013; Muñoz et al., Reference Muñoz, Cadierno and Casas2018; Verspoor et al., Reference Verspoor, de Bot, van Rein, De Houwer and Wilton2011). Peters (Reference Peters2018) found that more than 40% of the surveyed participants (i.e., Dutch-speaking adolescent L2 learners of English in Flanders) regularly watched English TV (with or without subtitles). Crucially, watching (subtitled) TV in out-of-school contexts was related to aspects of L2 development such as L2 learners’ receptive vocabulary knowledge. This was also the case in follow-up research in Flanders that found that watching TV promotes receptive (Puimège & Peters, Reference Puimège and Peters2019a) and productive vocabulary knowledge (Bollansée et al., Reference Bollansée, Puimège, Peters, Werner and Tegge2021) of L2 learners of English. González Fernández and Schmitt (Reference González Fernández and Schmitt2015) found that watching films, television, and video correlated with knowledge of certain collocations in English in adult Spanish learners of English. In addition, Lindgren and Muñoz (Reference Lindgren and Muñoz2013) found that watching subtitled films was not only one of the most common activities of primary school EFL learners in seven European countries, but it also had a strong impact on learners’ listening and reading skills. Similarly, De Wilde et al. (Reference De Wilde, Brysbaert and Eyckmans2020) showed that watching subtitled television correlated positively with L2 knowledge of 10–12-year-old children who did not yet have formal L2 instruction. However, it was not as important as social media, gaming, and speaking English in this particular age group. Kusyk and Sockett (Reference Kusyk and Sockett2012) reported that half of the students in their participant pool watched English television weekly by downloading or streaming TV series. Interestingly, it was found that frequent TV viewers had significantly more knowledge of frequently occurring multiword combinations than non-frequent viewers of English TV.

It should, however, be stressed that the abovementioned studies focused on L2 English; little is known about the use and effects of audio-visual input in out-of-school contexts for other target languages. One exception is the article by Peters et al. (Reference Peters, Noreillie, Heylen, Bulté and Desmet2019), in which they compared the impact of out-of-class exposure on Flemish L2 learners’ vocabulary knowledge in English and French. It was found that L2 learners’ exposure to French was generally limited to formal classroom-based instruction while they engage daily with (spoken) English language input through songs, TV, internet, and gaming. This was found to have an impact on participants’ vocabulary knowledge with significantly lower scores for French than for English. Drawing on the impact of input exposure in out-of-school-contexts, teachers should be encouraged to prepare and stimulate language learners to engage in out-of-class viewing and listening activities, especially for contexts where the L2 is not present in daily life and where exposure is limited to the L2 classroom, the teacher, and the textbook (Tragant et al., Reference Tragant, Muñoz and Spada2016).

Finally, since these viewing activities take place in out-of-class contexts, they may be particularly beneficial for autonomous or self-directed learners who can fully exploit or regulate their L2 viewing experiences. In this respect, Cole (2015; as referenced in Vanderplank, Reference Vanderplank, Dressman and Sadler2020) argued that autonomous learners were more focused on the linguistic elements of English language TV programmes than class-based learners.

7.1 Use of audio-visual input in the classroom

King (Reference King2002, p. 510) stated that audio-visual input ‘compensates for many of the shortcomings in the EFL experience by bringing language to life’. It has also been argued that video provides a kind of authentic and naturalistic exposure to spoken language that could not be realized by textbooks alone (Bednarek, Reference Bednarek2018; Jones & Horák, Reference Jones and Horák2014). However, little is known about the role of audio-visual input as part of a classroom activity or as part of a specific didactical approach.

Recently, audio-visual input has been used to operationalize input-based tasksFootnote ¹² that are used to provide a rich source of input in a task-based language teaching (TBLT) approach. In Lee and Révész (Reference Lee and Révész2020), for instance, short news clips of 20 to 50 seconds long were used that contained two target grammatical constructions (i.e., verbs in present perfect and past simple) and were either presented without captions, with captions, or with enhanced captions (i.e., the target construction is highlighted in the caption). Next, Korean ESL learners were asked to play the role of a newsroom editor. This meant that they had to watch the clips and decide, for each of the clips, whether the title and news category were appropriate. Next, they had to watch news clips in Korean and report to their friends what they saw in English in an oral and written production test.

So far, large-scale systematic investigations into the use of video in course materials and in teachers’ day-to-day practices are lacking. One exception is the chapter of Mariotti (Reference Mariotti, Gambier, Caimi and Mariotti2015) that discusses the use of video with on-screen text (i.e., captions, subtitles, reversed subtitles) in L2 teaching by 45 European foreign language teachers. She found that 39 out of the 45 surveyed teachers were familiar with audio-visual materials for L2 teaching. About one third of the surveyed participants said they used subtitled materialsFootnote ¹³ a lot while 25.6% used them sometimes. Teachers’ main reason for using subtitled audio-visual input is to improve students’ oral comprehension. They also considered audio-visual materials useful for increasing L2 learners’ motivation as well as their listening skills and vocabulary learning. In addition, Mariotti's findings revealed that teachers considered subtitles appropriate for beginning learners whereas captions were believed to be more appropriate for intermediate and advanced learners. Teachers also reported they used subtitled audio-visual materials for introducing L2 learners to course contents, to expose them to the L2 or in L2 tests. However, they also mentioned that they need support on how to use on-screen text effectively in the classroom.

Finally, there are also indications that TV-series are ‘underutilized’ in ESL Classrooms, as was argued by Bednarek (Reference Bednarek2018, p. 224) based on L2 learners’ responses to questions such as ‘my English teacher has used/shown examples from English-language TV series in the classroom’. Similarly, Vanderplank (Reference Vanderplank2016a) stated that the implementation of captioned video viewing in the classroom is still limited even though it is recognized as a valuable activity in informal out-of-school settings.

7.2 Choosing a type of on-screen text for viewing activities

This article has discussed the potential of audio-visual input for fostering different aspects of L2 learning. In addition, it has reviewed findings on the role of different types of on-screen text for supporting L2 learners and promoting L2 learning gains. Table 3 presents a tentative summary of research findings to schematically represent the potential language learning benefits that are associated with the different types of on-screen text. Two main findings seem to emerge from this table. First, different types of on-screen text may lead to different types of learning gains but most robust results have been found for captioned video. Second, most studies have looked into the potential of on-screen text for vocabulary learning.

Table 3.

Potential language learning benefits associated with the use of different types of on-screen text as compared to video only (without on-screen text)

While on-screen text may present numerous linguistic benefits, not all videos come with options to activate on-screen text. If teachers want to use on-screen text or give learners the possibility to activate on-screen text, this often means that they have to create materials themselves. However, this requires not only technical training (i.e., to use subtitling software and video editing tools) but also represents a serious time investment, especially when the input still needs to be transcribed/translated before on-screen text can be added. Luckily, technological advances and streaming services have made different types of on-screen text (e.g., captions, glossed captions) more easily available, which may contribute to the more widespread use of these types of materials inside and outside the classroom. We also find several dedicated evidence informed L2 learning tools. IdiomsTube (Lin, Reference Lin2021), for instance, is a computer-assisted learning tool that aims at promoting formulaic language learning from English YouTube videosFootnote ¹⁴ with captions. To do so, it automatically assesses video difficulty, generates pre-learning activities and tasks, and recommends videos. Another example is the ‘Language Reactor’ that offers different tools to promote L2 learning from authentic materials such as bilingual subtitles, captions with glosses, dictionaries, and so forth, for a wide range of Netflix series and YouTube videos, including different target languages.

To further transfer insights on the role of video and on-screen text from research to teaching practice, at least two aspects should be considered. The first consists in sharing research results with L2 teachers through practical sessions that may also involve technical training and worked-out scenarios with audio-visual input for L2 teaching. Second, it should be noted that publishers of learning materials (e.g., textbook publishers) can play an important role in selecting and adapting different types of audio-visual materials for classroom use to familiarize learners from the start with audio-visual input and different types of on-screen text.

7.3 Preparing learners in the classroom for out-of-class viewing

Webb (Reference Webb, Nunan and Richards2015) argued that one of the greatest benefits of audio-visual input is that it can provide large amounts of exposure to spoken L2 input because of its overall availability and accessibility. Audio-visual input therefore holds great potential for extensive viewing or the ‘regular silent uninterrupted viewing of L2 television inside and outside of the classroom’ (Webb, Reference Webb, Nunan and Richards2015). The concept of extensive viewing is based on previous research into the role of extensive reading programs (e.g., Cobb, Reference Cobb2007) that had revealed positive effects on, amongst others, comprehension (e.g., Nakanishi, Reference Nakanishi2015) and vocabulary learning (e.g., Webb & Chang, Reference Webb and Chang2015).

To promote regular out-of-class viewing, initial classroom-based viewing is required (Webb, Reference Webb, Nunan and Richards2015). L2 teachers thus play a crucial role in promoting learners’ use of audio-visual input and in establishing ‘a dialogue with learners about their viewing and listening habits’ (Sockett, Reference Sockett2012). In addition, they need to raise learners’ awareness (Webb, Reference Webb, Nunan and Richards2015) of the benefits of L2 audio-visual input and prepare them in the classroom for autonomous viewing. So far, research that has explicitly focused on in-classroom preparation for extensive viewing is lacking, but several questions could be addressed, including the following: How can we train teachers and learners in effective use of audio-visual input? What is the most effective way to promote use of audio-visual input in out-of-class contexts?

Even though research on this topic is lacking, there are at least three recommendations that can be formulated to prepare learners for out-of-class viewing activities.

Recommendation 1: Select appropriate material in function of the specific objective of the viewing activity (e.g., listening, comprehension, grammar, vocabulary).

The selection of appropriate materials for L2 learners is a difficult task.Footnote ¹⁵ However, this review has put forward several criteria that influence video comprehension as well as learning gains. These findings suggest that a careful selection of videos that is based on various parameters such as the lexical coverage (see section 2.3), the potential contribution of imagery, and the role that on-screen text may play for stimulating different learning goals (see Table 3 for a summary) can lead to significant learning. For instance, teachers might want to consider using captions to promote learners’ speech decoding and comprehension as well as to promote incidental vocabulary learning.

Recommendation 2: Raise learners’ awareness of the benefits of video by showing how it can support L2 learning.

This review has discussed different language learning benefits (e.g., vocabulary) that have been found after watching audio-visual input. While some of these benefits were found after relatively short exposure to audio-visual input, more substantial learning gains can occur when learners engage more extensively with video in out-of-classroom contexts. By drawing learners’ attention to gains that can occur through watching video in the classroom, we may raise learners’ awareness of the potential of video material for L2 learning, which is crucial if we want to stimulate learners to engage in independent out-of-class viewing.

Recommendation 3: Raise learners’ awareness of different types of on-screen text by using different options in the classroom and by explaining how they can support L2 learning.

To educate learners about the support that on-screen text can offer, we could teach them how to use these options. We could also highlight the advantages and drawbacks that have been associated with different types of on-screen text through different types of viewing activities in the classroom. For instance, subtitles might be helpful for beginning L2 learners to help increase their comprehension of authentic L2 input.