1. Introduction
Although the development of listening comprehension has long been neglected (Oxford, Reference Oxford1993), it is now considered one of the most important skills in L2 learning (Vandergrift, Reference Vandergrift2011) and “the primary means of L2 acquisition for most people” (Rost, Reference Rost2002: 103). At the same time, listening has been qualified as a highly complex skill because of its real-time (Buck, Reference Buck2001) and implicit nature (Hulstijn, Reference Hulstijn2007). Thus, listening success depends on a listener's capacity to simultaneously activate knowledge sources (e.g., speech decoding, word recognition, interpretation) and rapidly process the input.
The emergence of technology-enhanced language learning environments has triggered new ways of teaching listening, which are based on the use of authentic audiovisual material (Vandergrift, Reference Vandergrift2011). Yet, when using authentic input such as television programs for the teaching of L2 listening, it is important to know whether the input is manageable for the learner and which factors affect successful comprehension. This question was addressed by Webb (Reference Webb2010) and Webb and Rodgers (Reference Webb and Rodgers2009) who explored the importance of lexical coverage, i.e., the number of words needed for achieving adequate understanding. In their study, Webb and Rodgers analyzed the transcripts of 88 television programs of different genres (e.g., news, drama) and found that learners need a vocabulary size of 2,000 to 4,000 word families (including proper nouns and marginal words) for a 95% coverageFootnote 1, which is required for adequate comprehension (Nation, Reference Nation2006). The use of authentic input might, however, still be difficult even when learners have an acceptable vocabulary size. Factors such as the speed of the input and the presence of low-frequency words (Webb, Reference Webb2010) make listening a particularly challenging skill for L2 learners. Yet one of the advantages of using CALL and multimedia environments is that the environments might include different types of help options (Cárdenas-Claros & Gruba, Reference Cárdenas-Claros and Gruba2013) such as paced audio delivery and on-screen text. In the remainder of this paper, we focus on the potential of captions, i.e., same-language or intralingual subtitles, for supporting L2 learners’ comprehension of audiovisual documents.
Previous studies have shown that captioned video tends to be beneficial for improving L2 learners’ video comprehension (e.g., Baltova, Reference Baltova1999; Guillory, Reference Guillory1998). Captions may also stimulate vocabulary learning (e.g., Reference Montero Perez, Peters, Clarebout and DesmetMontero Perez, Peters, Clarebout, & Desmet, forthcoming; Sydorenko, Reference Sydorenko2010) because they help learners to isolate lexical items and pay attention to them (Vanderplank, Reference Vanderplank1990; Winke, Gass, & Sydorenko, Reference Winke, Gass and Sydorenko2010).
While extensive research has documented the benefits of captions, it has also been claimed that full captions require too much reading and therefore prevent the development of listening skills (King, Reference King2002). Keyword captions (i.e., only the key words of a sentence are captioned) might provide a solution to this problem, but their effectiveness for improving comprehension has been insufficiently researched (e.g., Guillory, Reference Guillory1998). Moreover, while research has focused on the linguistic benefits of captions, little is known about how learners themselves perceive the usefulness of (keyword) captioned video. In order to address both issues, this study investigates the effects of keyword-captioned, full-captioned, and non-captioned video on listening comprehension. The study also aims at documenting learners’ perceptions of the usefulness of (keyword) captioned video. In other words, this study investigates whether less textual density in the captioning area is more in terms of effectiveness and usefulness.
2. Background research
2.1 Full- and keyword-captioned video for listening comprehension
Ever since Price's (Reference Price1983) pioneering study, captioning has been studied extensively as a tool to improve L2 listeners’ comprehension of audiovisual input (e.g., Baltova, Reference Baltova1999; Winke et al., Reference Winke, Gass and Sydorenko2010). In order to measure captioning effectiveness, the bulk of research studies administered a comprehension test to both a control group (video only) and a captioning group (e.g., Markham, Reference Markham1989). As shown in Table 1, comprehension tests measured either global comprehension, or a combination of global and detailed understanding. Only one study (Guillory, Reference Guillory1998) explicitly focused on detailed comprehension. Most studies found that the captioning group significantly outperformed the control group on the listening test (see Table 1), which suggests that captions can improve comprehension (e.g., Chung, Reference Chung1999; Markham, Reference Markham2001). Moreover, captions may facilitate speech segmentation and decoding (Bird & Williams, Reference Bird and Williams2002; Garza, Reference Garza1991; Vanderplank, Reference Vanderplank1988) because they clearly visualize word boundaries. Captions therefore support learners’ aural word recognition (Winke et al., Reference Winke, Gass and Sydorenko2010) and might help them to “assign meaning to previously unintelligible aural entities, gradually building their aural comprehension in relation to their reading comprehension” (Garza, Reference Garza1991: 246).
Reviewed studies on full captions and listening comprehension
Note. Studies preceded by * found that the captioning group outperformed the no captioning control group, studies preceded by ? reported mixed results.
Although the linguistic benefits of captioned video have been documented extensively, it has also been claimed that captions hinder listening skill training rather than helping it (King, Reference King2002; Pujolà, Reference Pujolà2002; Vandergrift, Reference Vandergrift2004). Indeed, it is still a controversial issue whether learners provided with captions are training their listening or their reading skills (Pujolà, Reference Pujolà2002). It has also been argued that captions counteract with the development of compensatory skills and strategies (King, Reference King2002) which present a “significant aspect of L2 listening” (Buck, Reference Buck2001: 50) and are crucial for coping with authentic listening situations. A possible solution to this problem could be to provide learners with keyword captions (Garza, Reference Garza1991; Winke et al., Reference Winke, Gass and Sydorenko2010), i.e., same-language subtitles representing only key words of a sentence or paragraph, which decrease learners’ captioning dependency considerably. However, we are only aware of three studies that investigated the effectiveness of keyword captions for listening comprehension so far (Guillory, Reference Guillory1998; Reference Montero Perez, Peters, Clarebout and DesmetMontero Perez et al., forthcoming; Park, Reference Park2004).
Guillory's study (1998) involved beginning-level learners of French, who were assigned to three conditions: full captioning, keyword captioning, and no captioning. The keyword captions, determined by teaching experts, represented approximately 14% of the video transcript. Participants watched two clips twice before completing a listening comprehension test. Results revealed that the full captioning and keyword captioning groups significantly outperformed the control group. The difference between both captioning groups was not significant. Guillory concluded that keywords are not only as effective as full captions in supporting listening comprehension, they also substantially reduce text dependency.
Park (Reference Park2004) investigated the effectiveness of full and keyword captions for low intermediate, intermediate, and high intermediate ESL learners’ listening comprehension. He found that for high intermediate learners, keyword captions are as supportive as full captions, and helped them outperform a control group. For intermediate learners, however, the full captioning group significantly outperformed the keyword group, which in its turn outperformed the control group. For low intermediate learners, there was no difference between the keyword group and the no captioning control group, which were both outscored by the full captioning group. Park therefore concluded that only high intermediate learners benefit from keyword captions, which contradicts Guillory's (1998) conclusion. Yet, one potential limitation of this study concerns the fact that the same video input was used for the three proficiency levels. Although captions help learners to perform at a higher level (Neuman & Koskinen, Reference Neuman and Koskinen1992), the videos should match learners’ actual proficiency level (Danan, Reference Danan2004). It therefore remains unclear whether the keywords were ineffective or whether the videos were too challenging for the lower intermediate and intermediate learners.
Reference Montero Perez, Peters, Clarebout and DesmetMontero Perez et al. (forthcoming) compared the effectiveness of three captioning types (full captions, keyword captions, full captions with highlighted keywords) to a no captioning control group for video comprehension and incidental vocabulary learning (i.e., as a by-product of listening for meaning). Findings revealed that the four conditions scored equally well on the comprehension tests, which seems to contradict previous research results (e.g., Guillory, Reference Guillory1998; studies marked with * in Table 1). Montero Perez et al. hypothesized that this finding might be explained by the type of information targeted by the comprehension questions. Because they also studied the incidental learning of 17 target words, listening questions could only focus on parts of the clips that did not contain target words. While this was important to prevent an effect of the listening task on the word retention scores, it might have minimized differences between the four conditions on the comprehension tests.
Although a number of studies have compared full captions to keyword and no captions, it remains difficult to reach any firm conclusions on the benefits of keywords for improving comprehension.
2.2 Perceived usefulness of (keyword) captions
While the majority of studies focused on the linguistic benefits of captions, as measured by listening and vocabulary post tests, far fewer studies investigated how learners perceive the usefulness of captioning (Winke, et al., Reference Winke, Gass and Sydorenko2010). Pujolà (Reference Pujolà2002) found that learners’ perceptions of the objectives of captioning were associated with their listening proficiency. Whereas advanced learners tended to perceive captions “as a backup to their listening activity” (op. cit.: 254), lower-level learners considered captions indispensable for successful comprehension. Pujolà also concluded that learners’ perceptions might play a decisive role in their task approach and use of captioned video.
With regard to learners’ use, it has been found that learners interact less with captions than hypothesized (Grgurović & Hegelheimer, Reference Grgurović and Hegelheimer2007). Yet, although learners often tend to neglect help options in listening programs (Cárdenas Claros & Gruba, 2013), Winke et al.'s study (2010) revealed that learners perceive captions as a “crutch” (op. cit.: 81), which suggests that learners match what they hear with its visualization in the captions. Likewise, Vanderplank (Reference Vanderplank1988) qualified captions as a “mediating device” (op. cit.: 280). Sydorenko (Reference Sydorenko2010) reported that beginning learners considered video images more useful than captions. She hypothesized that learners tend to prefer the images because they only understand parts of the captions.
Guillory (Reference Guillory1998) analyzed the usefulness of keyword and full captions and found that 59% of the keyword group considered the keywords useful for improving comprehension, whereas only 23% of the full captioning group considered their support helpful. Park's data (2004) revealed that learners’ attitudes towards keyword captions varied according to their proficiency level. Whereas high intermediate learners were positive about the keywords and considered them challenging, intermediate and low intermediate learners found the keywords distracting.
So far, no study has thoroughly examined learners’ perceptions of captioning usefulness. Additionally, little is known about learners’ attitudes towards the usefulness of keyword captions. Yet, since perceptions might influence learners’ use (Pujolà, Reference Pujolà2002; Graham, Reference Graham2006), more research is required to find out why and under what circumstances captions are deemed useful.
3. Research questions
Based on the review of captioning literature, the following issues warrant further research:
(1) Only three studies so far have explored the effectiveness of keyword captions. In addition, they have produced mixed results. Therefore, this study investigates the effectiveness of two types of captioning, i.e., full captioning (FC) and keyword captioning (KC), compared to a no captioning (NC) control group, for listening comprehension.
(2) Far too little attention has been paid to how learners perceive (keyword) captions when watching L2 video. Therefore, the second part of our study focuses on a qualitative analysis of L2 learners’ perception of (keyword) captions.
The specific research questions are:
• Does the type of captioning have a differential effect on listening comprehension?
• How do L2 learners perceive the presence and usefulness of (keyword) captions for understanding L2 video and how do they think (keyword) captions help or hinder them?
4. Method
4.1 Participants
The participants were 226 undergraduate students (89 males, 137 females) at a Flemish university (political, social, communication, and economic sciences). All students (M age = 19.57, SD = 1.00) had an obligatory course in French, which focuses on communicative competence. Classes were organized in groups of approximately 25 students. Intact groups were assigned randomly to one of the three conditions: FC (N = 81), KC (N = 75), or NC (N = 70). The majority of the students reported Dutch as their first language (L1). Although four students reported Bulgarian, Russian, or Albanian as their mother tongue, their data were not excluded from the analysis because they were considered fluent speakers of Dutch. The data of six bilingual students (French-Dutch) were excluded from the study. Participants’ scores on a vocabulary size test (see section 5) indicated that they were intermediate learners of French.
4.2 Materials
4.2.1. Clip selection
We selected three authentic French television reports from a Swiss (clips 1 and 2) and a Belgian (clip 3) broadcasting house. The first clip (7'57”) dealt with the history, marketing strategy, and financial situation of the Lego© brick company. The second clip (5'44”) focused on the growing importance of private schools in Switzerland. The third clip (2'29”) presented a brewery in the North of France and its export strategy. All clips had a single narrator providing background information and included short interviews. Five experienced lecturers considered the difficulty level of the clips B1(+), according to the criteria of the Common European Framework of Reference (Conseil de l'Europe, 2011) and therefore appropriate for the intermediate-level participants. The lecturers thought the video images were supportive of the content yet not highly explicit.
We transcribed each video and added full or keyword captions with MAGpie.Footnote 2
4.2.2. Keyword determination
Keywords are defined as words that are important for the meaning of the sentence or paragraph. We used the same keyword determination procedure as Guillory (Reference Guillory1998) and Park (Reference Park2004) and contacted experienced lecturers of French. Five lecturers watched the clips, read the corresponding transcripts, and underlined the words they considered important for understanding the sentence or paragraph. We compared their selectionsFootnote 3 and determined a final set of keywords representing 14% of the entire transcript. Table 2 classifies the number of keywords per clip and per type. The keywords labelled multiword unit consist of more than two part of speech categories (e.g., collocations, combinations of nouns and adjectives) and are included as a whole in order to fully represent the key idea of the sentence or paragraph.
Overview of keywords per clip
Note. The multiword units contained fewer words in clip 2 than in clips 1 and 3, which explains the lower percentage of keywords for clip 2.
The keywords were synchronized with the audio and appeared centered in the captioning line, immediately before the word was presented in the audio. The duration of presentation depended on the type of keyword (single or multiword) and varied between one and two seconds (Merleau, 1981, as cited in Guillory, Reference Guillory1998).
4.3 Data collection instruments
4.3.1. Vocabulary size
We developed a vocabulary size test consisting of 49 multiple choice itemsFootnote 4, each containing four Dutch translation options. The test included three parts, corresponding to three frequency bands: 2,001 to 4,000 words (19 items), 4,001 to 5,000 words (16 items), and 5,001 to 7,000 words (14 items). Word frequency was based on the Routledge (Lonsdale & Le Bras, Reference Lonsdale and Le Bras2009), Verlinde (Selva, Verlinde & Binon, Reference Selva, Verlinde and Binon2002), and Dutch Parallel Corpus (Paulussen, Macken, Vandeweghe & Desmet, Reference Paulussen, Macken, Vandeweghe and Desmet2013) frequency lists.
Given the importance of vocabulary size for video understanding (Webb & Rodgers, Reference Webb and Rodgers2009), we included a vocabulary size test rather than a general proficiency test. Students’ scores on the vocabulary size test would serve as covariate in the statistical analyses.
4.3.2. Listening comprehension
We developed a comprehension test for each video. After seeing a clip once, learners completed the corresponding test. Together, the three tests consisted of 43 itemsFootnote 5 and included both receptive and productive items. As shown in Table 3, 29 items focused on the global ideas of the clips and the interviewees. The remaining items focused on detailed ideas. In order to answer the detailed questions, participants had access to the corresponding audio clip. So, for example, for a question on a topic discussed between time stamps 1'22’’ and 1'50’’, we gave the learner the corresponding audio clip (28 seconds). Students could listen to the clip once and write down the answer while listening. We opted for this approach because it allowed us to see if learners are able to identify the answer from the audio. Moreover, by providing the audio passage, we measured their actual comprehension rather than their capacity to memorize detailed information.
Characteristics of the test items for the 43-item comprehension test
All questions targeted the understanding of literal content. Although Buck's (2001: 114) “competency-based listening construct” also includes inferencing questions, we were unable to test learners’ understanding of implied meanings, because the three clips were relatively short documents which presented concrete and factual information.
All questions targeted passages that contained the keywords because (a) by focusing on keywords, the questions dealt with important information and irrelevant details were avoided; (b) if we included questions on parts that were not keyword captioned, we would not be able to see the difference between the NC and KC groups. In the example below, the words in italics were keyword captioned:
Question: What is « French Craftworks » and what is its objective ?
Answer: It is an association of French brewers. By joining forces, they aim to stimulate the export of French beer.
Students answered in their L1 (Dutch) to ensure that their L2 writing skills would not interfere (Buck, Reference Buck2001). Appendix A includes all comprehension questions.
4.3.3. Questionnaire
The final questionnaire consisted of six 6-point scale questions: two items on caption need (e.g., “If I had been able to activate captions, I would definitely have done that”), two items on (keyword) caption use (e.g., “I was distracted by the presence of (keyword) captioning and therefore focused less on the audio”), and two items on (keyword) caption usefulness (e.g., “I managed to understand the clips because (keyword) captioning was available”).
The open-ended survey questions asked all the participants whether they would activate full captions (with a definition of full captions) if they could choose and why they would or would not do so. Students in the KC group also completed a similar question on the usefulness of keyword captions. We only included the question on the usefulness of keywords in the questionnaire of the KC group. For the FC and NC groups, it would have been difficult to understand what keyword captions actually are since they did not watch such clips.
4.4 Procedure
One month prior to the learning session, we administered the vocabulary size test during regular classroom hours. Participants needed fifteen minutes to complete the test. Although the learning session itself did not follow their regular schedule, students were informed beforehand that their attendance would count as a regular class.
During the learning session, which took approximately 70 minutes, each student worked individually at a PC with a headset. All tests were paper-and-pencil tests. Before starting the videos, we informed participants that they were going to participate in a study on the use of video. We did not mention the main objective of the experiment, i.e., the effect of two captioning types. Participants received the following instruction: “You are going to watch three short clips once. After each clip, you will have to answer a number of comprehension questions. There will be global comprehension questions and detailed questions accompanied by audio.” We showed that the audio questions were preceded by the following sign on the tests: 
. The corresponding audio clips were available online, via a hyperlink mentioned on each listening test. Students were asked to listen to each audio clip once. At the end of the session, participants filled out the questionnaire.
4.5 Scoring
Answers to the comprehension questions were scored dichotomously: one point for a correct answer and zero for an incorrect one. The maximum score on an open-ended question depended on the information targeted by the item: if learners needed to provide three elements, one point was awarded to each correct answer. Partial credit was given when learners provided only a part of the expected answer. Appendix A indicates the scoring per question. Next, we calculated the total score of the 43 comprehension items, and the scores on the two subparts (global and detailed ideas). This approach was adopted because the experienced lecturers considered the difficulty level of the videos similar. Moreover, this allowed us to refine our understanding of when captions affect comprehension.
4.6 Data analysis
We analyzed the data using a combination of quantitative and qualitative data analysis methods.
Research question 1 (Does the type of captioning have a differential effect on listening comprehension?): We conducted analyses of covariance (vocabulary size score as covariate) and set the significance level in the statistical analyses at .05. We report partial eta squared (ηp2) as effect size, which explains the proportion of total variance of an effect, without the effects of other independent variables or interactions. Cohen's (Reference Cohen1988) rules of thumb are used for effect size interpretation: small, ηp2 > .0099, medium, ηp2 > .0588, and large, ηp2 > .1379.
Research question 2 (How do L2 learners perceive the presence and usefulness of (keyword) captions for understanding L2 video and how do they think (keyword) captions help or hinder them?): The analysis of the six-point scale is based on descriptive statistics. We used a conventional content-analysis approach (Hsieh & Shannon, Reference Hsieh and Shannon2005) to analyze the answers to the open-ended questions. This approach starts with the observation of the data before deriving appropriate coding schemes and coding the answers (see Results section).
5. Results
5.1 Vocabulary size
Table 4 lists the descriptive statistics of each condition per frequency band. A one-way ANOVA showed no difference in vocabulary size between the three conditions (total score): F (2, 223) = 0.93, p = .396, ηp2 = .008.
Descriptive statistics of vocabulary size per frequency band
The vocabulary size test had a good reliability (N = 226; Cronbach's alpha = .80) and was therefore used as a covariate in the comprehension analyses to control for learner differences.
5.2 Listening comprehension
Table 5 summarizes the mean scores on the comprehension test, as well as on the two subparts.
Mean scores and standard deviations on listening comprehension
We performed an ANCOVA with condition (type of caption) as between-subject variable, the total score on the listening test as dependent variable, and vocabulary size as covariate. We found a significant effect of condition: F (2, 222) = 6.07, p = .003, ηp2 = .05, and a significant effect of the covariate: F (1, 222) = 67.41, p < .001, ηp2 = .23. A Bonferroni post-hoc test revealed that the FC group significantly outperformed the KC and NC groups. We found no difference between KC and NC.
Next, we investigated whether the results were consistent in both subparts (general and detailed comprehension). We found that the results on the global comprehension questions significantly correlated with the results on the detailed comprehension questions (r = .461, p < .001) and therefore performed a MANCOVA analysis (Tacq, Reference Tacq1997).
Results showed that condition significantly affected the test scores: F (4, 442) = 3.01, p = .018, ηp2 = .03, as well as the covariate: F (2, 221) = 39.28, p < .001. The covariate showed a very large effect: ηp2 = .26.
The analysis revealed that condition had a significant effect on the scores of questions on the main ideas, but not on the detailed ideas. A significant relation was found between vocabulary size and the two components measured (see Table 6). The positive b-values indicate a positive relationship between test scores and vocabulary size. A Bonferroni post-hoc test revealed that FC significantly outperformed the KC and NC groups on global comprehension; no difference was found between KC and NC.
MANCOVA of global and detailed comprehension
5.3 Questionnaire
The second research question focused on L2 learners’ perception of the usefulness of (keyword) captions when watching L2 video.
Table 7 lists the descriptive statistics for the six questionnaire items. The results showed that the NC group seemed rather confident about understanding the clips without support (item 1). However, the average score on item 2 indicates that they would have activated captioning if it had been available.
Descriptive statistics on need and usefulness
Note. Students who did not fill out the questionnaire were excluded from the analysis.
The average score on item 3 shows that the FC and KC groups did not find the (keyword) captions very distracting. But learners in the KC group reported an average score of 3.63 out of 6 on item 4, which suggests that they did not always read the captions.
Quantitative analyses of items 5 and 6 showed that learners in the FC group found the full captions relatively useful for clip understanding and test completion, with average scores higher than 4.60 on the 6-point scale (with 6 as most useful). The KC group was less positive, with average scores between 3 and 4 on the 6-point scale and therefore considered the keywords only slightly useful.
5.4 Content analysis of open-ended questions
The first open-ended question asked whether learners (all groups) would activate full captioning if they could choose and why they thought captioning would or would not help them. We performed a content-analysis on learners’ responses, which was based on three consecutive reading and coding cycles. After a first reading, different themes emerged from the data. We grouped these themes into categories which reflect theoretical constructs: i.e., perceptual processing and meaning-making processes (Anderson, Reference Anderson1995; Rost, Reference Rost2002) and findings from empirical research: vocabulary acquisition (Reference Montero Perez, Peters, Clarebout and DesmetMontero Perez, et al., forthcoming). We also added two learner-related categories, i.e., learner competence and learner comfort. We coded all answers in a second reading. A third reading allowed us to discriminate between specific reasons per coding category. We then performed frequency counts per category and reason. For example, we added the answer of participant KC-1-108 (see below) to the category perceptual processing. Next, we found that the answer referred to both reasons included in the perceptual processing category (see Table 8). When an answer of a single participant referred to multiple reasons, we included the answer in each corresponding category or reason. A few participants did not give a specific reason.
Perceived usefulness of full captioning
Results showed that learners’ high score on the perceived usefulness of full captions (see Table 7, items 5 and 6) is confirmed in the analysis of the open-ended question: 175 out of 226 participants would activate captions.
Students indicated that they found captions useful for reasons related to the perceptual processing of the speech stream:
“Yes I would activate captioning, they speak so fast, and it's easier to see the words apart from each other, because now one word fades into another.” (Participant/KC-1-108)
The majority of the participants mentioned the usefulness of captions for meaning construction:
“I would activate full captions to make sure that I understand what is said and to retrieve information out of it.” (Participant/KC-1-100)
Other students explicitly referred to the advantages of bimodal input:
“I would definitely use full captions because then, the message enters via two channels and that is easier to understand.” (Participant/FC-1-113)
In addition, 42 participants considered captions useful for reasons of learner comfort:
“Yes I would use captions, as a matter of convenience. For example, at home, I would use captions because you need to focus less in order to understand the meaning.” (Participant/NC-1-004)
On the other hand, 23 participants experienced captions as distracting:
“No, because then you will no longer listen but only read what is being said.” (Participant/KC-1-171)
We also asked participants in the KC group whether they would use keyword captions if available (see Table 9). We used the same analysis and coding procedure as for the question on full captions. Surprisingly, only 17 out of 75 participants would activate keywords:
Perceived usefulness of keyword captioning
“It was the first time I saw this, and I think it was very comprehensible and useful, especially to remember some numbers and keywords.” (Participant/KC-2-134)
Two students thought that keywords were useful for training their listening skills:
“Yes, I would use them. If the aim of the exercise is to understand the content by listening, keywords suffice.” (Participant/KC-2-140)
The majority of the KC group thought, however, that captions were distracting:
-“No, I would not choose keyword captioning, full captioning is easier. I focus too much on the keywords and then I miss the rest of what is said.” (Participant/KC-2-108)
-“No, too distracting because you have to switch constantly between reading and listening.” (Participant/KC-2-175)
The answers tend to suggest that keywords draw learners’ attention, yet the participants seemed to have problems in managing the combination of reading keywords at the same time as listening.
6. Discussion
6.1 Effectiveness of (keyword) captions
(1) With regard to learners’ understanding of global ideas, the quantitative analysis revealed that the FC group significantly outperformed the NC and KC groups. We found no differences between NC and KC. Results of our study are thus consistent with the majority of studies on full captions in which the beneficial role of full captions is generally acknowledged (cf. Table 1).
However, we should stress that the mean scores on the comprehension questions were rather low. The poor results may be attributed to the fact that learners were only allowed to watch the clips once and had no “second encounter” (Pujolà, Reference Pujolà2002: 241). It is also possible that certain parts of the videos were too challenging for the learners because of low frequency collocations such as frôler le naufrage (to be on the verge of disaster) and gravir les echelons (to be promoted). Although our questions did not explicitly target these structures, learners might have noticed the unknown words and tried to discover their meaning. By doing so, they might have missed other, more important parts of the clips. This was also suggested by Goh (Reference Goh2000) who studied L2 listeners’ difficulties and noted that “neglect the next part when thinking about meaning” (op. cit.: 63) was the fourth most cited problem.
Since we found no difference between the KC and NC groups, which were both outperformed by the FC group, our results are in line with Park's (2004) findings for lower-intermediate learners, and therefore contradict Guillory's (1998) conclusion. In the data collection section, we showed that the listening questions only focused on the parts that were also presented in the keyword captioning, and therefore important for the meaning. Yet our findings tend to suggest that either keywords failed to direct learners’ attention to those elements or learners were unable to adequately interpret the meaning. Results of our second research question may clarify why learners in the KC group did not outperform NC.
(2) A quantitative analysis of the detailed questions with audio revealed no differences between the three groups. Whereas we found a small effect size of condition on this part of the test, we found a very large effect for vocabulary size. A plausible explanation for this finding is that the audio clip made the participants rely most on their vocabulary size because there was no support (in the form of on-screen text or visual images) available while processing the audio input.
What needs to be clarified as well is why the FC group did not outperform the other groups. Previous research suggested that captions help learners to get the “aural cues” (Garza, Reference Garza1991: 246) the next time the same input is heard. Because captions help aural word recognition (Winke et al., Reference Winke, Gass and Sydorenko2010), learners’ decoding load might decrease and provide them with more time to allocate attention to other processes (Pulido, Reference Pulido2007) such as meaning interpretation. Our results, however, suggest that the relationship between decreased decoding effort and interpretation is not self-evident in this context. It is thus possible that learners in the FC group recognized the words better even though it did not result in better understanding of the meaning of the message (Goh, Reference Goh2000). However, we are unable to check this hypothesis because we only included a comprehension test.
6.2 Perceived usefulness of (keyword) captioning
6.2.1. Full captioning
The qualitative analyses showed that the majority of learners would choose to activate full captions. Learners’ need for full captions is strong and might be triggered by their lack of confidence in their listening ability (Graham, Reference Graham2006; Pujolà, Reference Pujolà2002).
Learners’ main reason for wanting to activate captions is at the level of meaning construction (121 participants). Indeed, most learners feel that they would achieve better comprehension with full captions. A number of students also explicitly stated that bimodal input is easier to understand than video only. The second most frequently mentioned reason concerned perceptual processing issues: students reported that captions might support speech segmentation, but also help them to manage the fast speech rate or disambiguate unclear pronunciation. We should also notice that learners who were provided with captions (the FC group) mentioned the perceptual processing reasons more than the other two groups.
Based on students’ self-reports, Goh (Reference Goh2000) and Graham (Reference Graham2006) found that L2 listeners’ problems are mainly at those levels. Consequently, it is not surprising that learners perceived captions to be most useful for dealing with perceptual processing and meaning-making problems. Thus, the results with regard to learners’ perception of the usefulness of full captions indirectly corroborate Goh's (2000) and Graham's (2006) findings on listening difficulties and also align with Pujolà's findings for lower-level listeners (2002). Moreover, participants’ perception of usefulness unambiguously reflects the results of earlier empirical studies, in which captions were shown to improve comprehension (e.g., Chung, Reference Chung1999; Markham, Reference Markham2001) and help speech stream decoding (e.g., Bird & Williams, Reference Bird and Williams2002; Vanderplank, Reference Vanderplank1988).
The results of our questionnaire also suggest that watching video with captions is considered more relaxing than video without captions because participants consider their reading skills better developed than their listening skills. This aligns with Graham (Reference Graham2006) who found that learners tend to consider their listening skills their least developed L2 skill. Because learners can rely on the captions, their listening anxiety (Vanderplank, Reference Vanderplank1988) might be reduced, leading to a better listening performance and more confidence in their listening skills (Graham, Reference Graham2006).
A minority of participants (23 out of 226) indicated that they would not activate captions. One of their reasons was that they considered captions harmful for their listening training. It is important to point out that the majority of those participants were from the NC group. Only a few students considered captioning distracting. Other students suggested that the input was clear enough or that they would only activate captions for parts that were unclear to them. Although they responded rather negatively to this question, they did consider captions useful but not necessary in every context. It might therefore be interesting to give learners the opportunity to turn captions on and off, according to their specific needs.
6.2.2. Keyword captioning
The content-analysis of the KC group's responses revealed an explicitly negative attitude towards the use of keyword captions. This finding did not, however, emerge from the questionnaire data included in Table 7 (e.g., item 3). Only 17 out of 75 students would activate keyword captions if available and they reported mixed reasons for doing so: better comprehension, recall of words and numbers, keywords help but do not hinder listening training, keywords help to focus, and are better than nothing. An analysis of the negative results revealed that none of the reasons referred to perceptual and meaning-making processes. Rather, learners stressed that keywords are highly distracting and hamper the listening activity. A plausible explanation for this response might be the salient nature of keywords. Although keywords did not enhance recall of content, keywords might have drawn attention to particular elements. Indeed, learners reported that the salient and irregular appearance of the keywords is precisely what hindered them in hearing the rest of the video. Thus, keywords might have caused listening problems rather than supporting the listening process. This interpretation might also explain why participants indicated that they did not always read the keyword captions (see Table 7, item 4). Finally, the findings of the qualitative data elucidate the findings of our quantitative data: the content analysis suggests that the KC group could not outperform the NC group because the keywords were distracting rather than helpful.
7. Concluding remarks
This study focused on the effectiveness of two captioning techniques for listening comprehension and the perceived usefulness of (keyword) captioned video.
Our findings revealed that: (1) full captioning helps learners to improve global comprehension when watching an L2 short-length video clip once, (2) keyword captions do not lead to better comprehension than no captions, (3) learners’ vocabulary size is significantly related to their comprehension. The qualitative data showed that learners highlight the usefulness of captions for different linguistic and learner-related reasons. In contrast, keyword captions are generally described as distracting for learners’ listening experience.
In sum, results of the quantitative and qualitative analyses seem to reject the hypothesis that less textual density is more in the context of L2 listening comprehension.
These findings have two pedagogical implications. First, when teachers want to use videos for listening tasks, they should be encouraged to use full captions because they enhance learners’ global comprehension and are considered useful by the L2 learners. Second, our findings also corroborate the need to help learners enlarge their vocabulary to ascertain that they have sufficient lexical coverage to gain adequate comprehension of the L2 video input.
However, there are a number of limitations in this study. A first limitation concerns the difficulty level of the video materials. Although the clips were considered appropriate for our participants, a number of low frequency words might have made certain parts too challenging. A detailed analysis of the transcripts based on word frequency and lexical coverage could clarify (e.g., Webb, Reference Webb2010). However, this was beyond the scope of this study. Second, the type of clips used did not allow us to measure L2 learners’ understanding of inferred meaning and therefore prevents us from drawing a more complete picture of captioning effectiveness.
We conclude this paper with two future research tracks. First, studies might examine the effectiveness of captions for improving comprehension of full-length TV programs (Rodgers, Reference Rodgers2012) rather than short clips. Studies might also investigate whether systematic and long-term exposure affects learners’ comprehension, attitudes, and perceptions of captions. Second, it might be interesting to study learners’ reading behavior and attention allocation while watching captioned video. By means of eye-tracking technology, we could analyze these questions in an objectified way.
Further insights into the overall functioning of captioning will undoubtedly shed new light on the potential of captioning as a pedagogical tool and therefore open new paths for researching its value.
Supplementary material
To view supplementary material for this article, please visit http://dx.doi.org/10.1017/S0958344013000256
