1. Introduction
Captioned videos have been extensively employed by instructors and learners as a multimedia scaffold to enhance the understanding of second language (L2) oral discourse. Captions – visual text delivered in the same language as the audio – hold the potential to help L2 learners segment and parse running speech during listening (Leveridge & Yang, Reference Leveridge and Yang2013, Reference Leveridge and Yang2014). Notably, a meta-analysis (Montero Perez, Noortgate & Desmet, Reference Montero Perez, Noortgate and Desmet2013) illuminated that multimodal captioned videos may exert a selective effect on L2 learners with different cognitive profiles (e.g. beginning, intermediate, and advanced). In this vein, both Mayer (Reference Mayer2001) and Kozan, Erçetin and Richardson (Reference Kozan, Erçetin and Richardson2015) maintained that whether technology-assisted multimodal materials can be beneficial to learners depends on two cognitive variables: (1) the amount of information listeners can hold at a given time (i.e. working memory capacity) and (2) listeners’ modality preferences (e.g. visual vs. auditory) while processing information delivered from different modalities. These two variables have been identified as major determinants affecting the success and failure in many aspects of L2 input processing (e.g. processing of reading materials: Kozan et al., Reference Kozan, Erçetin and Richardson2015; Sagarra, Reference Sagarra2017; processing of oral discourse: Révész, Reference Révész2012). Hitherto, these two factors have not been simultaneously examined in studies on multimodal L2 listening.
Moreno and Mayer (Reference Moreno and Mayer2007) posited that with multimodal input (e.g. oral discourse and captions) at play, processing demands may overload learners’ working memory. In this regard, how efficient L2 learners process multimodal input will determine the extent to which they will benefit from multimodal input. It is plausible that L2 learners with better working memory capacity may benefit more from the addition of extra input – captions – while viewing videos. It is also conceivable that the constraint of working memory capacity may force L2 listeners to be selective in their attention allocation (Mayer, Reference Mayer and Mayer2005), attending only to the information delivered in their most preferred modality (e.g. visual vs. audio). In this case, those who prefer auditory input may not consistently attend to captions, and those who prefer visual input may be more likely to allocate their attention to captions (Lee, Liu & Tseng, Reference Lee, Liu and Tseng2019). Drawing on eye-tracking technology, Lee and Révész (Reference Lee and Révész2018) found that L2 learners who were more caption-reliant spent more time fixating on visual input (such as captioned text) and exhibited greater learning gains after viewing captioned videos. This finding entails that L2 learners’ modality preference may affect their attentional allocation while viewing the video content, which in turn may yield differential gains (see Seufert, Schütze & Brünken, Reference Seufert, Schütze and Brünken2009; Sueyoshi & Hardison, Reference Sueyoshi and Hardison2005). Accordingly, it is plausible that working memory capacity and modality preference may take effect in tandem to influence L2 learners’ learning outcomes in a multimodal video viewing environment. To establish this, two research questions are investigated in this study:
1. Does working memory capacity modulate the effect of full captions on L2 listening outcomes? If so, to what extent?
2. Does preferred input processing modality (i.e. visual and auditory) modulate the effect of full captions on L2 listening outcomes? If so, to what extent?
2. Literature review
2.1 Empirical studies of captioning on L2 listening
L2 research has found that captioning helps learners visualize word boundaries and hence makes the indistinct auditory text more comprehensible (e.g. Chang, Tseng & Tseng, Reference Chang, Tseng and Tseng2011; Garza, Reference Garza1991; Huang & Eskey, Reference Huang and Eskey1999). Garza (Reference Garza1991) used full captions in videos to investigate whether advanced L2 learners of Russian and English would achieve higher listening comprehension. The results showed that the full captioning group significantly outperformed its control counterparts in the content-based comprehension test. Similar positive evidence is also found in Huang and Eskey’s (Reference Huang and Eskey1999) study. This study showed that intermediate L2 learners who watched a television series with full captions scored significantly higher in the listening comprehension test than that of their control counterparts. Winke, Gass and Sydorenko (Reference Winke, Gass and Sydorenko2010) asked L2 Spanish learners to watch three selected videos twice either with or without full captions. A significant difference was found between the full captioning and no captioning groups, suggesting that the exposure to full captions substantially aided the understanding of audiovisual materials (see also Markham & Peter, Reference Markham and Peter2003; Markham, Peter & McCarthy, Reference Markham, Peter and McCarthy2001; Mirzaei, Meshgi, Akita & Kawahara, Reference Mirzaei, Meshgi, Akita and Kawahara2017). Similarly, Peters (Reference Peters2019) also observed that full captioning significantly enhances L2 learners’ listening comprehension. In addition to its effectiveness in promoting L2 listening outcomes, Chen, Liu and Todd (Reference Chen, Liu and Todd2018) and Pujadas and Muñoz (Reference Pujadas and Muñoz2019) both provided positive empirical evidence supporting the potential of full captions for the acquisition of vocabulary forms (see also Suárez & Gesa, Reference Suárez and Gesa2019).
Despite the above positive evidence, a few studies have suggested that the effectiveness of full captions may be modulated by L2 learners’ modality preference (visual vs. auditory; see Lee, Liu & Tseng, Reference Lee, Liu and Tseng2019) and individual differences in preference for visual input (i.e. captions; Lee & Révész, Reference Lee and Révész2018). This view is corroborated by Montero Perez et al.’s (Reference Montero Perez, Noortgate and Desmet2013) meta-analysis of 15 L2 captions studies. Although Montero Perez et al. found different effect sizes for learners of different linguistic profiles (beginning: g = 0.66, p > .05; intermediate: g = 1.55, p < .001; advanced: g = 0.71, p > .05), they argued that this result should be “interpreted with care given the fact that only one study involved advanced participants,” and, importantly, the medium effect size for beginning learners was not significant (p. 732). Montero Perez et al. noted that “we found no [significant] difference in the effect of captioning between the proficiency levels for listening comprehension,” and this led them to conclude that “the relationship between captioning effectiveness and proficiency level may be less problematic than outlined in primary research studies” (p. 732). Accordingly, Montero Perez et al. maintained that individual differences in cognitive profiles, such as learners’ working memory capacity and modality preferences, may play a more prominent role than differences in linguistic profiles in modulating the effectiveness of captioning.
2.2 Working memory and its potential role in modulating the effect of captions
Linck, Osthus, Koeth and Bunting’s (Reference Linck, Osthus, Koeth and Bunting2014) meta-analysis of 79 caption studies showed that working memory is positively associated with many aspects of L2 processing and proficiency outcomes, with a small-to-medium effect size (p) of .255. This entails that working memory capacity is associated with L2 learning outcomes. In particular, in a multimedia- or technology-assisted L2 learning environment, learners have to process information from different modalities presented to them simultaneously. Processing oral discourse and other video content is already an excessively loaded event; having to process additional input (captions) would probably impose more cognitive load on L2 learners – and this is probably why the L2 learners in Taylor’s (Reference Taylor2005) study found full captions distracting.
However, the above hypothesis may not be true if L2 learners’ input processing in a technology-assisted learning environment could be non-linear in nature. In non-linear input processing, learners may rely on all available cues, albeit to different extents probably due to differential preferences for different modalities. While all of the available cue/input may contribute to understanding, L2 learners may swiftly switch back and forth between and among cue/input presented from different modalities at different points in time. They therefore constantly engage, disengage, and shift attention between and among input from different modalities. This, according to Mayer and Moreno (Reference Mayer and Moreno2003), has the potential to offload the cognitive load of relying only on (a) particular channel(s) (i.e. focusing only on the oral discourse). In this regard, working memory capacity would not be excessively loaded or drained even when an extra visual multimedia support (captions) is being added. Consequently, individual working memory differences may not significantly mediate the effect of captioning on listening comprehension.
2.3 Preferred modality and its potential role in modulating the effect of captions
Researchers have employed retrospective instruments such as questionnaires or self-reports (e.g. Reid, Reference Reid1995) to make inferences about L2 learners’ modality preferences. However, offline self-reported data from offline retrospective measurements may not accurately reflect learners’ preferred modality in real-time input processing (Leveridge & Yang, Reference Leveridge and Yang2013, Reference Leveridge and Yang2014; Winke, Gass & Sydorenko, Reference Winke, Gass and Sydorenko2013). Specifically, post-study retrospection can allow a researcher to accurately report on one’s modality preference in real-time video processing only when and if swift processing of video content is reliably open to consciousness. However, as seen in Lee et al. (Reference Lee, Liu and Tseng2019), L2 learners’ modality preference while viewing the video material is not always consistent with their offline verbal report conducted after the video viewing task. Lee et al. found that notwithstanding L2 learners’ superior performance under the partial captioning viewing condition – an indication of their preference for partial captioning while viewing the video materials – they found captioning distracting in the post-study retrospective questionnaire. Thus, time-sensitive instruments that are capable of tapping into L2 learners’ reliance on captions while viewing video content are needed in order for researchers to gain insights into L2 learners’ modality preference in real-time input processing.
The Caption Reliance Test (CRT) developed by Leveridge and Yang (Reference Leveridge and Yang2013, Reference Leveridge and Yang2014) is a time-sensitive assessment originally developed to assess English as a foreign language (EFL) high school learners’ reliance on captions while viewing captioned videos. The CRT has the potency to probe learners’ preferred modality (visual vs. auditory) in real-time input processing by asking them to swiftly respond to manipulated (visual vs. auditory) input without (too much) thinking (see the Method section for more details). Using the CRT, Leveridge and Yang (Reference Leveridge and Yang2013, Reference Leveridge and Yang2014) found that lower-level learners rely more on full captions than their higher-level counterparts. They further found that EFL learners with a heavier reliance on captions (visual learners) tended to score lower in their listening comprehension tests. This study demonstrates the intimate relationship between modality reliance and multimodal L2 listening. Recently, drawing on the CRT, Lee et al. (Reference Lee, Liu and Tseng2019) examined whether learners’ modality preference while viewing captioned videos affected their listening outcomes. Lee et al. observed that due to different modality preference (visual vs. auditory), L2 learners showed differential reliance on captioned texts; this determined the extent to which L2 learners may have benefited from a captioning viewing environment (see also Hasler, Kersten & Sweller, Reference Hasler, Kersten and Sweller2007). Finally, using eye-tracking technology, Lee and Révész (Reference Lee and Révész2018) observed that visual L2 learners showed significantly longer gaze duration time fixating on captioned texts, and this exerted a significant impact on their gains in L2 grammar. However, whether different modality preferences (i.e. visual vs. auditory) affect L2 learners’ listening outcomes was not directly explored in existing studies.
3. Method
3.1 Participants
The participants originally consisted of 84 Chinese-speaking college-level learners of English from Taiwan. Note that 24 participants’ data were not included in the analysis because their (mediocre) performance did not meet the screening thresholds of the instruments used in this study (see section 3.4.2). The 60 participants were all psychology majors and required to read English-medium publications in their field of study. Their average age was 21.5, and they had learned EFL for 12 years in instructional classroom settings where classroom teaching still focused primarily on grammar. All participants were comparable not only in terms of their overall English proficiency, but also their proficiency in all four language skills; they attained a high-intermediate (i.e. B2 CEFR) level of competence in their overall score and in all subsections in a standardized English proficiency test (i.e. TOEFL, TOEIC, IELTS). All of the participants had extensive experience using multimedia materials such as captioned videos for various language learning purposes.
3.2 Materials
An 11-minute TED Talk video on successful leadership was selected as the experimental material for this study. TED Talk videos are one of the most widely used L2 listening materials used by college EFL learners in many Asian (EFL) countries. The video used in this study was selected based on the criteria suggested in Mirzaei et al. (Reference Mirzaei, Meshgi, Akita and Kawahara2017): (1) linguistic difficulty level, (2) accent, and (3) topic familiarity. First, the linguistic difficulty of the video was compatible with the participants’ actual proficiency level, as determined by four EFL instructors (Mirzaei et al., Reference Mirzaei, Meshgi, Akita and Kawahara2017; Montero Perez et al., Reference Montero Perez, Noortgate and Desmet2013). Second, the speaker’s accent in the video is standard American English, the most common accent used in the participants’ English learning experiences. Third, the topic of the video was familiar to the participants.
3.3 Design
A 2 × 2 × 2 factorial design was used to explore the impact of two within-subject independent variables (working memory capacity and modality preferences) and one between-subject independent variable (full vs. no captions) on one dependent variable (listening comprehension scores). A randomized block design (see Figure 1) was employed to ensure group comparability prior to the study. Specifically, the grouping procedure began with all participants receiving (1) a CRT to measure their preferred input modality (visual vs. auditory) and (2) a reading-span task (RST) to determine their working memory capacity (high vs. low; see the Instruments section). Based on CRT results, all participants were distributed into visual and auditory groups. Next, all visual and auditory group members took the RST; according to the RST scores sequenced in ascending order (lowest to the highest), a number was assigned to each associated participant (e.g. the participant with the lowest RST score is number “1,” the second lowest is number “2,” etc.). Once a number was assigned to each participant, the odd numbers were put into the no captioning group, and the even numbers were put into the captioning group. This grouping process ensured no significant difference between the full captioning and no captioning groups in terms of their RST score distribution. Importantly, to make inferences about the modulating role working memory capacity played in this study, the participants were dichotomously distributed into two groups (i.e. high and low) based on a 50% percentile split procedure.
A visual schematization of the grouping procedure
Under both experimental and control conditions, the participants received a listening comprehension test to assess their understanding (see the Instruments section). A questionnaire was distributed at the end of the study to collect information regarding the participants’ perceptions of the task.
3.4 Instruments
The instruments that were employed in this study were (a) the CRT, (b) an RST, (c) a listening comprehension test, and (d) a post-study questionnaire. The Corpus of Contemporary American English was used to determine that the frequency of the words used in the instruments were level appropriate for the participants in this study.
3.4.1 The CRT
The CRT consisted of 41 items. Each item involved two successive components: (1) watching a level-appropriate captioned video about a short dialogue between a man and a woman and (2) answering a multiple-choice comprehension question. Based on Leveridge and Yang (Reference Leveridge and Yang2013, Reference Leveridge and Yang2014), the CRT should consist of congruent and incongruent items. In the congruent items, the audio texts and captions in the video were identical; in the incongruent items, there was a one-word mismatch between what the participants heard and saw:
(The audio and captions from the dialogue below are simultaneously presented)
Man: Some of my students just won’t pay attention in class.
Woman: It might help if you introduce medication to them (what learners hear).
It might help if you introduce meditation to them (what learners see).
After viewing the video, the participants had to answer a multiple-choice comprehension question and were explicitly asked to listen for the correct answer:
Question: What can be the man’s solution to his problem?
(A) meditation (B) regulation (C) medication (D) recitation
To accurately assess preferred modality online, it was imperative that the participants were unaware of the incongruent items during the CRT. To prevent this, this study adopted a 75% congruent and 25% incongruent ratio (31 vs. 10 items, respectively) based on the guideline prescribed by Leveridge and Yang (Reference Leveridge and Yang2013, Reference Leveridge and Yang2014). The participants’ performance on the 10 incongruent items was the basis for our determination of the participants’ modality preference. Following Leveridge and Yang (Reference Leveridge and Yang2013, Reference Leveridge and Yang2014), it was assumed that visual learners were more likely to attend heavily to captions and thus gave answers based on what they saw. In this respect, they were more likely to choose (A) from the example above. Auditory learners, on the other hand, may be more likely to select (C), for they preferred to rely more on input presented in sound form. However, the participants were not told that if they noticed any discrepancies between the audio and the visual text, the audio was to be assumed correct. The reason being that this instruction may prime the participants about the nature/purpose of the task.
Based on the scoring policy used in Liu and Todd (Reference Liu and Todd2014) – another study that also utilized the CRT – one point was awarded under the “visual modality” for choosing (A), and one point was awarded under the “auditory modality” for choosing (C). After totaling the points for either modality, each participant’s preferred modality (PM) index value was calculated by dividing the visual modality points by the auditory modality points (see Table 1).
Caption Reliance Test score ballot sheet
The participants relying more on visual input, such as captions, achieved a PM index value larger than 1 (e.g. 7 visual modality points/2 auditory modality points = PM index value > 1), whereas those attending more to the audio texts achieved a PM index value of less than 1 (e.g. 2 visual modality points/7 auditory modality points = PM index value < 1). This categorization, consequently, divided participants into two groups: (1) visual learners (PM value > 1) and (2) auditory learners (PM value < 1). The validity and reliability of the CRT was stringently verified using the item analysis theory (IRT) model and multivariate discriminant analysis.
3.4.2 The RST
The RST – one of the most extensively used working memory measures in many L2 and caption studies (e.g. Alptekin & Erçetin, Reference Alptekin and Erçetin2009; Gass, Winke, Isbell & Ahn, Reference Gass, Winke, Isbell and Ahn2019) – was employed in this study. The RST used in this study was based on Friedman and Miyake (Reference Friedman and Miyake2004). Typically administered on a computer, the RST has 12 trials in total. Each trial features a sentence set consisting of two to five sentences. The RST used in this study included three trials consisting of two sentences, three trials consisting of three sentences, three trials consisting of four sentences, and three trials consisting of five sentences. Accordingly, there were 42 sentences in 12 trials. All the sentences were comparable in length (sentence length range: 12–17 words, 20–22 syllables, 55 to 73 letters). The maximum presentation time for each sentence was 6.5 seconds. All sentence sets were randomly presented to the participants on a computer screen to reduce the order effects.
As far as the procedure of the RST is concerned, following Turner and Engle (Reference Turner and Engle1989), each RST item began with the participants reading a sentence on the projector screen (e.g. “Working memory is important for reasoning and the guidance of decision-making and behavior.”). Then the participants had to quickly determine whether the sentence they just read made sense or not by answering (aloud) yes or no. Immediately after responding yes or no, an isolated letter appeared after a 1-second delay, and the participants had to retain the letter in their minds for later recall. After repeating the above procedure for all the sentences in a given trial, a question mark (see Figure 2) appeared on the screen to prompt the participants’ recall of all the letters presented in a trial. A point was awarded to a participant if the letter that followed a given sentence was successfully recalled at the end of a trial. Because there were 42 letter recalls in the 12 trials, the highest point for the RST was 42 points. Based on their performance on the RST, the participants with scores below the 50th percentile were grouped into the “low” working memory group, and those with scores above the 50th percentile were placed into the “high” working memory group.
The procedure for a trial in the RST
The first trial ended when the participants indicated that they had finished recalling all the letters before proceeding to the second trial. A paper-based instrument was used to collect all of the responses in this study. The entire task (12 trials) took roughly 20 minutes to complete.
To calculate the RST scores, we followed Friedman and Miyake (Reference Friedman and Miyake2004) by adding the number of isolated letters correctly recalled across all trials (one point for each accurate recalled letter). Asking the participants to judge sentence validity tapped into the processing function of working memory and asking the participants to recall the isolated letter tapped into the storage function. Thus, the participants engaged in both storing (remembering isolated letters) and processing (understanding the sentences) – two critical functions of working memory – during the RST. In particular, for sentence verification, this study adopted an 80% accuracy criterion from Turner and Engle (Reference Turner and Engle1989); those with a total score below the 80% accuracy rate (n = 24) were excluded from the analysis, as this suggested that the participants might be exclusively focusing on the isolated letters without devoting their attentional resources to processing sentences.
3.4.3 The listening comprehension test
The comprehension test contained 15 test items to assess participants’ global, local, and inferential understanding of the TED Talk video. Each item consisted of four options with only one correct answer. The participants heard and read the stem of each comprehension question on the screen. They then read the options on the screen before writing their answers on the answer sheet. Each correct answer was given one point; accordingly, the highest score was 15 for 15 test items. The validity and reliability of the comprehension test were also examined using the IRT model and multivariate discriminant analysis.
3.4.4 Questionnaire
The questionnaire administered at the end of the study aimed to elicit participants’ general experience of watching captioned videos and perception of watching the captioned video in this study.
The questionnaire consisted of two parts: two 5-point Likert-scaled statements aiming to probe the participants’ multimodal listening experience while viewing the video and one open-ended comment allowing them to provide their qualitative comments on their viewing experience. The two statements are:
Item 1: I focused more on the audio than the captions while watching the video.
Item 2: I focused more on the captions than the audio while watching the video.
While the full-captioned L2 learners had to answer both the two Likert-scaled statements and the open-ended comment, the no-captioned L2 learners only had to respond to the open-ended item asking about their viewing experience.
3.5 Procedure
During the first meeting, all participants were grouped into two viewing conditions based on their CRT and RST scores (see Figure 1). It took the participants approximately 30 minutes to complete the CRT and RST. One week after the CRT and RST tasks, each group was separately invited to take part in an 11-minute TED Talk video viewing session. After viewing the video, a listening comprehension test, which took about 10 minutes to complete, was immediately administered to assess their understanding of the viewing material. After taking the comprehension task, the participants completed the post-study questionnaire, which was completed by all participants within 1 minute.
4. Data analysis
The current investigation is a 2 × 2 × 2 factorial design study that delves into the impact of three between-subject independent variables (working memory capacity vs. modality preferences vs. captioning viewing mode) on one dependent variable (listening comprehension scores). Based on the CRT and RST results, the participants were dichotomously placed into “high vs. low working memory” and “visual vs. auditory” groups; these four groups of participants were randomly assigned to either a full- or no-captions viewing condition. To address the two research questions of this study, these aforementioned experimental variables were entered and translated into a three-way ANOVA. The questionnaire results, which aim to contextualize and substantiate the findings of ANOVA, were analyzed using simple descriptive statistics (see section 5.3).
5. Results
5.1 Descriptive statistics
Table 2 offers an overview of the participants’ listening comprehension scores vis-à-vis the variables of this study. There exists a significant variance in their listening performance under the experimental conditions of this study (as gleaned from the values of standard deviation shown in Table 2). This indicates that individual differences (i.e. PM, working memory capacity, or both) probably have modulated the effects of captions on the participants’ performance, and this calls for a more fine-grained ANOVA analysis.
Descriptive statistics of the participants’ listening comprehension test scores
Note. CM = caption mode; PM = preferred modality; WM = working memory; NC = no caption; FC = full caption; A = auditory; V = visual; L = low working memory; H = high working memory.
5.2 Three-way ANOVA
The ANOVA (see Table 3) revealed that the participants’ PM significantly affected their listening comprehension scores (p = .02). This significance was found to have a medium-to-large effect size (η2 = .09). Notably, the auditory learners significantly outperformed the visual learners in their listening comprehension outcomes, as Figure 3 visually schematizes such contrast.
The three-way ANOVA result
Note. CM = caption mode; PM = preferred modality; WM = working memory.
Mean difference between auditory and visual L2 learners’ listening scores
The listening advantage of the auditory learners, however, did not always hold true when taking caption mode into consideration. As shown in Table 3, there was an interaction (p = .03, η 2 = .08) between the participants’ PM and their caption viewing conditions (i.e. full caption or no caption) on their listening performances. Post-hoc analysis from an independent t-test further confirmed that captions exerted a significantly disruptive effect on the auditory learners (p = .048). The disruptive effect was visually schematized in Figure 4, showing that when exposed to full captions, auditory learners performed substantively poorer than visual learners than without captions. However, without captions, the listening advantage for the auditory learners resurfaced, as they significantly outperformed their visual counterparts (p = .00) in the no-captioning condition. Despite the negative effect of captions imposed on the auditory learners, such an effect was not observed in the visual learners’ listening comprehension under full- and no-caption conditions (p = .144).
The interaction between caption mode and preferred modality
As shown in Table 3, the effect of captioning did not reveal a significant effect on visual learners’ performance (M NC = 10.28, M FC = 11.07, p > .05), and this suggests that other covariates such as PM as well as interaction effects between covariates explained more significant variance of listening comprehension performance. Notably, ANOVA detected an interaction among caption mode, PM, and working memory capacity (p = .01), with a close-to-large effect size (η2 = .10). Figure 5 shows that under the low working memory condition, captioning exerted neither a facilitative nor disruptive effect on both auditory and visual learners. That is, regardless of PM, the presence of captions did not significantly affect the listening comprehension of the L2 learners with lower working memory capacity.
The interactions among caption mode, preferred modality, and working memory capacity
Interestingly, a significant interaction between caption and modality preference conditions was observed for the L2 learners with high working memory capacity. Overall, the auditory learners with higher working memory outperformed their visual counterparts under the no-captioning condition (p = .001). However, when being exposed to full captions, the auditory learners’ listening advantage was dampened, as they scored significantly lower than those under the no-captioning condition (p = .021). This result reinforces the detrimental effect captions had on the auditory learners’ understanding, particularly those with higher working memory capacity.
Although captions failed to benefit the auditory learners with higher working memory capacity, it exerted facilitative effects on their visual counterparts. Figure 5 illustrates that for the visual learners with higher working memory capacity, full captioning significantly augmented their listening comprehension (p = .04) – a phenomenon not observed in the visual learners’ performance data when working memory capacity was not considered. When working memory capacity was considered, it accentuated the effect of full captions. This phenomenon suggests that visual learners with higher working memory capacity may better process multimodal information with (full) captions. The above findings establish the important role of working memory in determining the effect of PM in affecting the listening outcome under different viewing conditions (i.e. full and no captions).
5.3 Questionnaire data
The participants’ responses to the questionnaire are presented in Table 4.
Descriptive statistics of the questionnaire data with 5-point Likert-scale items (1 = “strongly disagree”; 5 = “strongly agree”)
Note. FC = full captions; V = visual learners; A = auditory learners; H = high working memory capacity.
5.3.1 Auditory learners under the full-captioning condition
In response to Item 1 (“I focused more on the audio than the captions while watching the video.”), the auditory learners’ rating is 4.6, indicating that they tended to strongly agree that their focal attention was directed to the oral discourse, particularly for those with high working memory capacity (4.93). Despite this, when responding to Item 2 (“I focused more on the captions than the audio.”), the auditory learners’ average rating is 3.79. The observation that auditory learners tended to give a higher rating for Item 1 (which focuses on the audio input) over Item 2 (which focuses on the captions) – 4.6 vs. 3.79 – is consistent with the corresponding results of the CRT (having a higher preference for the auditory input). Notably, despite their (auditory) modality preference, these auditory learners also gave a relatively high rating for Item 2 (e.g. 3.79). This suggests that these auditory learners – in particular the auditory learners with higher working memory capacity (3.87) – were drawn to read captioned texts, and they were all aware of their deliberate reading of captions.
The above information collectively indicates that captioning is a very strong attention getter: it has the potency to “force” the auditory learners to read captions in spite of their general inclination for the audio input. This is further supported by the comments from the auditory learners: two thirds of the auditory learners (8/15) from the full-captioning condition expressed that captions consumed a lot of their attention, four of whom even showed their strong dislike toward captions. When asked for their viewing preference, 10 auditory learners under the full-captioning condition preferred not having captions on. Among these 10 people, seven indicated that captions could direct their attention undesirably; two found captions distracting; and one stressed that captions could impose more cognitive load to their understanding of the multimodal information. The following are a few examples:
“… Captions can attract a lot of attention from me, and I don’t like it. When this happens, I feel like I’m not polishing my listening skills. I was paying a lot of attention on captions during video watching …”
“… Captions can be distracting …”
“… I felt captions could attract a lot of my attention during video watching …”
“… Captions can attract a lot of attention, and I don’t really like my attention being controlled by captions …”
“… Even though I’d normally turn captions on, I don’t like my attention being controlled by captions …”
“… But with captions, it can impose more cognitive load because you have to constantly attend to so many things. It can also be a little distracting …”
In contrast, visual learners’ average rating for Item 1 is 2.41, indicating that they tended to disagree that they attended to the oral discourse while viewing the captioned video; even for those visual learners with high working memory capacity, we see the same disagreement (2.7). When responding to Item 2, the visual learners’ average rating is 4.47, suggesting that they tended to strongly agree that they attended to captions; this tendency is even more manifest for the visual learners with high working memory capacity (4.87).
5.3.2 Visual learners under the full-captioning condition
None of the visual learners expressed a negative perception toward captions. Thirteen visual learners indicated that “captions helped me better understand the oral discourse in the video”: six visual learners claimed that captions would not be distracting; five regarded captions as facilitative to their understanding of the audiovisual cues; and two stressed that captions made them feel secure, allowing them to verify whether the received input is accurate. The following are a few examples:
“… I don’t find captions distracting, it can help me understand difficult terms …”
“… Watching videos with captions is much easier (less cognitive load) for me, whereas viewing videos without captions is more demanding/challenging to me.”
“… Captions allow me to make sure that what I listen is correct. I really enjoy having captions because it allows me to verify whether my listening is correct …”
“When I turn captions on, it’s less cognitively challenging to me because I can just read them. Watching videos without captions is more challenging because I have to force myself to concentrate and not to space out.”
Notably, five visual learners seemed to treat captions as an optional asset, namely a multimodal input that they could draw on when needed:
“… I attend to captions when I feel the need to extract details to facilitate my comprehension …”
“Captions can be helpful for understanding the content. If I don’t want to look at it, I just ignore it.”
5.3.3 Auditory learners under the no-captioning condition
Seven auditory learners specifically indicated that captions may direct their attention to specifics, rather than the tenor of the content. In that case, all of them preferred not having captions on during online viewing, for they were concerned that captions were not conducive to their listening comprehension:
“… Captions may be helpful for understanding some technical terms. But I wouldn’t turn captions on because it’s more focusing on understanding main ideas …”
“… I think if we were to train our listening comprehension skills, having captions on may be less desirable …”
“… Captions can draw a lot of attention and maybe this is not helpful for training L2 listening …”
5.3.4 Visual learners under the no-captioning condition
Twelve visual learners stated that they found captions attention demanding but not distracting. Importantly, five of them explicitly noted, “I typically find captions helpful in promoting my comprehension, but viewing non-captioned videos (i.e. the TED Talk video) also does not bother me at all because the video content already provides a lot of contextual visual clues.” Notably, these comments all came from the visual learners with high working memory capacity.
6. Discussion
6.1 Effects of caption modes on auditory learners
The quantitative data revealed that overall auditory learners performed better under the no-captioning condition. Furthermore, auditory learners’ advantage over visual learners was more salient under the no-caption condition. Therefore, it became clear that L2 auditory learners understood the audiovisual materials best when immersed in the no-captioned multimodal environment.
The prominent no-captioned effect manifested in the auditory learners may be accounted for by Hasler et al.’s (Reference Hasler, Kersten and Sweller2007) account. Hasler et al. maintained that in order to generate the most desirable learning outcome in a multimodal environment, what is preferred has to match with what is presented. In the case of auditory learners, their preferred processing channel was through what they heard rather than what they saw. However, when exposed to the (full)-captioning video environment, despite their unfavorable preference with the visual input, the auditory learners – in particular those with high working memory – still involuntarily attended to captions (as indicated in their responses to Likert-scaled items). In this case, the presence of full captions may have divided the auditory learners’ attentional resources (split attention). In contrast, the observation that having no captions turned out to be the optimal viewing condition for the auditory learners indicates that visual support (in this case, full captions) disrupted rather than facilitated auditory learners’ understanding of multimodal video content.
Both the quantitative and qualitative data indicate that the presence of full captions did not lead to the optimal multimodal processing environment for the auditory learners and lend empirical support for the need to recalibrate the multimodal training materials suitable for those who preferred to “listen” in a technology-enhanced environment.
6.2 Effects of caption modes on visual learners
Notwithstanding the notable effect captions had on the auditory learners, it did not significantly impact visual learners’ performance. This study found that the listening scores from the visual leaners under the full-captioning condition did not significantly differ from those under the no-captioning condition. Without the assistance of full captions, visual learners were able to achieve a similar level of understanding as those assisted with the visualized text. This illuminating finding indicated that for visual learners, full captions may not be the only scaffold facilitative to their listening comprehension.
Since visual learners tend to rely on what they see, it is possible that even without captions, they would still efficiently draw on other available elements presented visually – as indicated by the qualitative comments from the visual learners under the no-caption condition. Nonlinguistic or paralinguistic visual input (e.g. facial expressions, body language, etc.), for example, may contain subtle but rich contributing contextual cues that provide assistance equal to the effect of full captions in enhancing L2 listening outcomes. As evidenced in Sueyoshi and Hardison’s (Reference Sueyoshi and Hardison2005) study, paralinguistic elements, such as gestures and facial cues, can assist the understanding of videotaped lectures – a type of video content similar to the TED video used in this study. This may explain that even without captions, the L2 visual learners in this study were still able to comprehend at comparable depth with their captioned counterparts in a multimodal listening environment. Accordingly, visual learners – especially those with higher working memory capacity – are probably better able to make good use of available linguistic, nonlinguistic or paralinguistic visual input than auditory learners, irrespective of having access to (full) captions or not (see also Seufert et al., Reference Seufert, Schütze and Brünken2009).
Additionally, unlike the auditory learners whose qualitative data exhibited a strong tendency in rejecting the use of captions, the visual learners presented data from two different stances: (1) absolute endorsement: perceiving captions in a positive lens in all cases; and (2) conditional endorsement: perceiving captions as an optional asset at their disposal, which could be turned off and drawn on when necessary. It is important to note that the conditional endorsement all came from visual learners with high working memory capacity. This is suggestive of the selective effect of other intra-learner factors at work – that is, working memory capacity.
6.3 Effects of caption modes on L2 learners with lower working memory capacity
ANOVA revealed that although working memory as a factor alone did not modulate the effect of full captioning, there is a significant interaction among working memory, L2 learners’ modality predilection, and different caption modes. This interaction suggests that working memory significantly interacted with learners’ PM in modulating the effect of captions (the presence of full captions or the lack thereof) on the participants’ listening performance. The effect of captions was not observed in the L2 learners with lower working memory capacity – irrespective of their PM. This finding resonates with Mayer (Reference Mayer2001, Reference Mayer and Mayer2005), in which he posited that although multimedia-enhanced learning has the potency to enhance L2 learning outcomes, its promising effect is conditioned to the limited capacity of working memory.
And this is exactly what was found in this study: for the L2 learners with lower working memory capacity, full captioning exerted neither a facilitative nor disruptive effect on both the auditory and visual learners in this study; in this case, more (addition of full captions) may not be merrier (better comprehension). With limited capacity to simultaneously store and process (engage and disengage, activate and deactivate) ongoing information during video watching, multimodal input may have excessively loaded L2 learners with lower working memory capacity. In this regard, they may not have been able to effectively allocate attention to utilize the enriched multimodal input while watching the video, and the presence of captions thus made little difference to either visual or auditory L2 learners’ listening comprehension. They may therefore have found it difficult to take full advantage of multimodal information, including captions.
6.4 Effects of caption modes on L2 learners with higher working memory capacity
Both the auditory and visual learners with higher working memory performed better under their preferred input. Specifically, the visual learners significantly benefited from the captioned video (where they obtained an additional facilitative visual support, full captions), and the auditory learners performed most ideally without captions (where they could focus their advantageous attentional resources primarily on the oral discourse). Importantly, the discrepancy between the auditory and visual learners indicates that having higher working memory did not necessarily mean that they would equally benefit from multimodal input. Individual processing modality preferences needed to be considered in order to provide them with the “right” listening training materials.
Specifically, visual learners, when having sufficient attentional resources and more working memory capacity at their disposal, were more likely to benefit from the provision of full captions. The visual learners in this study reported that captions were not distracting or cognitively demanding; several of them even further indicated that they could voluntarily control the timing to refer to full captions, as well as other nonlinguistic or paralinguistic visual clues. It is worth noting from the qualitative comments that unlike the visual learners (in particular those with high working memory capacity), who could choose to dismiss full captions and use them when needed, the auditory learners with higher working memory capacity seemed to be less adept at harnessing the multimedia visual support. Specifically, as shown in the auditory learners’ Likert-scale responses, despite their dislike of captions, many of them found that they could not voluntarily dismiss reading captions while viewing the video. This indicates that (full) captions is an effective attention (re)directing device, and when being used on L2 auditory learners, it could disrupt rather than facilitate their parsing and understanding of oral discourse.
7. Pedagogical implications
This study showed that the effect of full captions was significantly affected by L2 learners’ cognitive profiles, namely modality preference and working memory capacity. The findings of this study suggested that instructors should be sensitive to L2 learners’ cognitive profiles when using captions to enhance their listening outcomes of audiovisual video input. As evidenced in this study, individual cognitive factors were found to be highly crucial in determining the effect of full captions. Language instructors are thus encouraged to use the assessments of modality preference and working memory capacity (i.e. the CRT and RST) to diagnose L2 learners prior to exposing them to a multimodal learning environment. Online L2 learning platforms may also want to provide such assessment tools, allowing learners to understand their cognitive profiles and use the multimodal video materials based on their profiles for optimal outcomes.
As evidenced in this study, those with average or above-average working memory capacity should be more aware of their real-time modality preference when using captioned video. In this group of L2 learners, captioned videos were not recommended for those who preferred to listen, but they were beneficial for those who were visually inclined. For the auditory L2 learners with better working memory capacity, videos with full captions may not have been a desirable listening environment, as the presence of full captions may have imposed an extra load to their information processing. Auditory L2 learners may also be advised that processing preference plays a huge role in affecting their general understanding when receiving multimodal input. Aligning with this principle, their visual counterparts may be informed that due to having more cognitive/attentional capacity in utilizing visual input, they are more likely to make better use of captions to assist their multimodal comprehension.
As for L2 learners with below-average working memory capacity, they should be informed that their modality preference does not affect the extent to which they benefit from full captions. Also, full captions may not provide overt assistance to their listening comprehension. Nonetheless, qualitative data in this study showed that there is an attitudinal discrepancy between the visual and the auditory learners’ listening preference. With the participants’ qualitative data suggesting that the visual learners were more inclined to refer to full captions and auditory learners were more resistant toward full captions, this may be a referential point for instructors when assigning multimodal training materials for lower working memory L2 learners.
8. Conclusion
Although this study yielded interesting insights on the use of L2 captioned videos instruction in enhancing L2 listening, three major limitations should be noted. First, the study used a TED Talk video as the experimental material. It has yet to be established whether the findings of this study also apply to video materials that include different types of visual cues and dynamic images such as TV shows or animations. Second, because this study focused on listening comprehension, the findings may not be generalized to other aspects of L2 development, such as the acquisition of L2 spoken vocabulary. Whether individual cognitive factors continue to play a prominent role in other aspects of L2 domains warrants further investigations. Third, this study only targeted the full-captioning condition, the most widely used captioning option in the literature. It is important to note that there are other captioning options such as real-time and partial captioning. Future studies may consider these captioning options and see if the findings of this study still hold in these different captioning viewing environments.
Ethical statement
The participants were informed about the experiment process and voluntarily agreed to participate without any duress. The participants clearly understood their right to withdraw at any time, and their consent was obtained before research got under way. The primary research protocol, which took the form of a video-viewing session, did not cause any harm to the participants. Confidentiality of data is ensured by identifying participants by their ID numbers. The authors declare that there is no conflict of interest.
About the authors
Emily Fen Kam is a PhD student in the Department of English at National Taiwan Normal University. Her research focuses on the listening/speaking development of foreign language learners.
Yeu-Ting Liu is a professor in the Department of English at National Taiwan Normal University. His research focuses on the cognitive aspects of second language acquisition. He can be contacted at National Taiwan Normal University, No. 162, Sec. 1, HePing E. Rd., Taipei 106, Taiwan. Email: yeutingliu@ntnu.edu.tw. All correspondence regarding this publication should be addressed to him.
Wen-Ta Tseng is an associate professor in the Department of Applied Foreign Languages at National Taiwan University of Science and Technology. His research focuses on language assessment.
Author ORCIDs
Emily Fen Kam, https://orcid.org/0000-0001-8631-5342
Yeu-Ting Liu, https://orcid.org/0000-0001-8055-0587
Wen-Ta Tseng, https://orcid.org/0000-0002-6946-0406
