1 Introduction
Central to development of second or foreign language (L2) listening skills is exposure to authentic and comprehensible input (Vanderplank, Reference Vanderplank2010). With the advances of technologies, authentic audio-visual materials such as broadcast news, movies and academic lectures have become easily accessible and increasingly embedded into language learning classrooms. While these resources provide rich content and reflect real-life language, they often entail complex listening comprehension skills (Gilmore, Reference Gilmore2007). To assist L2 listeners in comprehending these materials, facilitative tools such as visual or textual clues (Danan, Reference Danan2004) and speed controllers (Zhao, Reference Zhao1997) have emerged to enhance the pedagogical effect of this medium (Vandergrift, Reference Vandergrift2011).
One such tool is captioning which was first used to help hearing-impaired people when watching television, but later became popular as a medium for L2 instruction (Garza, Reference Garza1991). Following this, many researchers investigated whether captioning could improve language processing ability for L2 learners who are “hard of listening” as well as the “hard of hearing” (Vanderplank, Reference Vanderplank1988: 272). Many studies have highlighted the potential of full captioning not only to facilitate comprehension but also to promote vocabulary acquisition (Winke, Gass & Sydorenko, Reference Winke, Gass and Sydorenko2010; Montero Perez, Van Den Noortgate & Desmet, 2013). However, the conventional full-captioning method has also received critical attention for bringing too much textual assistance, promoting reading over listening and impeding the development of listening strategies (King, Reference King2002; Pujolà, Reference Pujolà2002; Vandergrift, Reference Vandergrift2004).
In the light of limitations of full captioning, keyword captioning emerged as a solution (Guillory, Reference Guillory1998). In this method, only keywords, which are manually selected by experts, are presented in the caption. Keyword captioning has its own limitations. The manual selection of keywords is not only costly and time-consuming but also subjective, which may not be beneficial when having a wide range of learners with different requirements. Furthermore, the irregular appearance of keywords on the screen has been found to distract the listeners (Montero Perez, Peters & Desmet, Reference Montero Perez, Peters and Desmet2014a).
To address these issues, we propose a novel method of captioning, partial and synchronizedFootnote 1 , which strives to effectively remedy the shortcomings of the conventional full and keyword captioning methods. This tool is meant to facilitate the comprehension process while encouraging listening to the audio more than reading the caption. In this method keywords are not the selection criteria. Instead, inspired by significant research on L2 listening difficulties, the selection is based on the major factors that impede listening comprehension (see Section 4). This method has two features: “partialization” and “word-level synchronization”. The former automatically reduces the amount of text and makes a selection of words, based on certain principles, to appear on the screen, while the latter forces each word to appear on the screen based on its timestamp (i.e. one-by-one, from left to right, in synchrony with the speaker’s utterance). Partialization promotes listening to the audio and referring to the caption only when encountering a problem. Meanwhile, synchronization emulates the speech flow, allows for text-to-speech mapping (Bailly & Barbour, Reference Bailly and Barbour2011) and avoids the irregular appearance of the words (Montero Perez et al., Reference Montero Perez, Peters and Desmet2014a). As a noteworthy feature of PSC, both partialization and synchronization processes are done automatically so that the method can be applied to a large number of video documents. Further, this feature allows the captions to be adjusted to the learners’ proficiency level.
The present research therefore has two main contributions: (1) proposing PSC as a new captioning tool to train L2 listening skills; and (2) introducing the criteria for selecting words in PSC to realize this method. Throughout the study we describe how the use of ASR technology and computational linguistics together with other components frames this system and enables the automatic generation of PSC. The study also reports on the evaluation of this new method compared with full captioning and investigates potential for its improvement.
2 Background
2.1 Full captioning and L2 listening comprehension
Full captioning is defined as visual text delivered along with video via multimedia, where the language of the written text matches the spoken content (Markham & Peter, Reference Markham and Peter2003; Leveridge & Yang, Reference Leveridge and Yang2013). Without being affected by accent, pronunciation or audio deficiencies, full captions enable the listeners to parse the speech stream into meaningful chunks (Garza, Reference Garza1991) – an essential learning process (Ellis, Reference Ellis2003). Full captioning also helps with the phonological visualization of audio to make listeners more certain of ambiguous input (Bird & Williams, Reference Bird and Williams2002).
A considerable amount of research has been published on the beneficial effects of full captioning. Some studies have investigated the effect of this method on word learning (Danan, Reference Danan1992; Montero Perez et al., Reference Montero Perez, Van den Noortgate and Desmet2013), reading development (Markham & Peter, Reference Markham and Peter2003), word recognition (Bird & Williams, Reference Bird and Williams2002) and listening comprehension (Taylor, Reference Taylor2005; Winke et al., Reference Winke, Gass and Sydorenko2010). A recent meta-analysis (Montero Perez et al., Reference Montero Perez, Van den Noortgate and Desmet2013) investigated the overall effect of full-captioned video on listening comprehension and vocabulary learning based on eighteen studies. The findings revealed a large effect of full captioning on listening comprehension (Hedges’ g=0.99) and vocabulary acquisition (g=0.87).
Most of these findings are derived from dual coding theory, which posits that using both verbal and visual inputs, learners can construct referential connections and thus learn more efficiently (Paivio, Reference Paivio1990). However, each of the sound and visual channels has limited capacity and can manage limited amounts of processing at a time (Mayer & Moreno, Reference Mayer and Moreno2003). This assumption is central to cognitive load theory (Sweller, Reference Sweller1994) and working memory theory (Baddeley, Reference Baddeley1992). Although these theories have not been widely researched in the context of L2 (Diao, Chandler & Sweller, Reference Diao, Chandler and Sweller2007; Mayer, Lee & Peebles, Reference Mayer, Lee and Peebles2014), some studies indicate that when authentic videos are accompanied with captions, learners’ differences in and limitations of cognitive capacity may impose difficulty in attending to the three sorts of input, i.e. audio, video and text (Taylor, Reference Taylor2005).In this respect, some learners may apply their attention selectively and prioritize reading over listening (Lund, Reference Lund1991; Pujolà, Reference Pujolà2002), since it may be hard to divide their attention equally between both skills (Chang, Reference Chang2009). In such cases, many L2 learners may find the audio difficult to process quickly and therefore use the captions as “the most understood, relevant and thus preferred stimuli” (Leveridge & Yang, Reference Leveridge and Yang2013: 202). In line with this, in a study by Sydorenko (Reference Sydorenko2010), who examined the effect of input modality and learners’ attention to the input, participants reported that they paid most attention to captions, then video, then audio. While the combination of different input is beneficial for L2 learners in many ways (such as vocabulary learning, comprehension, form recognition), when the goal is to train listening skills, full captions may not be the optimal tool (Diao et al., Reference Diao, Chandler and Sweller2007), hence the input should be enhanced to fit this purpose. Meanwhile, different types of captions (e.g. verbatim or keyword) can influence listening in different ways (Garza, Reference Garza1991). Accordingly, other methods such as keyword captioning and word-level or phoneme-level synchronized captioning have gained instructional value.
2.2 Keyword captioning
Guillory (Reference Guillory1998) examined the use of keyword captioning for beginning learners of French and reported that students who received keyword captions preformed as well as those who received full captions. In her study, keywords were selected manually based on their importance to the main idea of the videos. Guillory argued that “learners no longer need to be subjected to a volume of text to read; they can in fact comprehend authentic video with considerably less pedagogical support” (p. 95). She concluded that keyword captioning decreases cognitive load, improves multichannel processing and motivates learners to listen more, but read less.
Montero Perez et al. (Reference Montero Perez, Peters and Desmet2014a) investigated the effectiveness of keyword captioning and found no significant difference between the scores of the keyword captioning and no-captioning groups in the comprehension question. Yet they also investigated the perceived usefulness of keyword captioning and reported that this method was highly distracting. The researchers argued that the salient and irregular appearance of keywords on the screen could lead to such distraction. Note that the appearance of keywords is not synched to their utterances (cf. Section 2.4 and Table 1 for more information on the pros and cons of this method).
Comparison of different captioning methods
On vocabulary acquisition, however, another study showed that keywords were as helpful as full captioning (Montero Perez, Peters, Clarebout & Desmet, Reference Montero Perez, Peters, Clarebout and Desmet2014b). Here, full captioning, keyword captioning and full captioning with highlighted keywords were compared against the no-captioning condition. The findings revealed that the captioning groups scored equally well on form recognition and clip association and significantly outperformed the no-captioning group.
2.3 Word-level or phoneme-level synchronized captioning
Advances in automatic speech recognition (ASR) technology have enabled automatic text-to-speech alignment, leading to the development of word-level or phoneme-level synchronized captioning (Braunschweiler, Gales & Buchholz, Reference Braunschweiler, Gales and Buchholz2010). Here, synchronized captioning shows the words or phonemes in the caption one-by-one, which is realized by mapping each word or phoneme to its corresponding speech segment. In this sense, the major difference between this method and the full-captioning method lies in the synchronization unit (see Figure 1).
Screen shot for a TED talk with word-level synchronized caption (top) and conventional full caption (bottom) for consecutive time frames (t1–t4)
In the full-captioning method, synchronization is carried out at sentence level so that a chunk of words appears on the screen, remains there for a few seconds, disappears, and then the next chunk appears. On the other hand, word-level synchronized captioning is sequential word-by-word captioning in which the emergence of words on the screen is concurrent with the speaker’s utterance. In other words, instead of chunks of words appearing together, words appear one after another, from left to right, and in precise synchrony, as the speaker says each word. Synchronization at phoneme-level is the same except that the alignment unit is set to phonemes instead of words.
Examples of speech-to-text correspondences can be found in audio books, albeit in a different context. In this context, text-to-audio alignment is used to facilitate reading and text tracking. The Talking Books Project (Medwell, Reference Medwell1998) is an early example in which the text is read while each word is highlighted as spoken. These books were found to be beneficial for assisting children in text-to-speech mapping, word decoding and first language (L1) reading development (Medwell, Reference Medwell1998). Interactive electronic books are enhanced versions used to assist word reading and story comprehension (Korat, Reference Korat2010). Likewise, digital books, which use ASR technology to align each spoken word with the text, can potentially help L2 learners in reading development and vocabulary learning (Trancoso, Serralheiro, Viana, Caseiro & Mascarenhas, Reference Trancoso, Serralheiro, Viana, Caseiro and Mascarenhas2007). Bailly and Barbour (Reference Bailly and Barbour2011) experimentally investigated the use of a “karaoke-style” reading system for learning sound-to-letter mapping. This system allowed learners to select any segment of the audio and listen to it as the phonemes were highlighted incrementally. A cursor was set to move smoothly in real time under the phonemes that were being vocalized. In this study, the subjects’ scores on a word spelling test were compared using two different audio + text reading systems: (1) synchronous; (2) non-synchronous. The study suggested that the multi-modality of synchronous reading implicitly facilitated text-to-speech mapping in French for native and non-native participants.
As found in prior research (Medwell, Reference Medwell1998; Trancoso et al., Reference Trancoso, Serralheiro, Viana, Caseiro and Mascarenhas2007; Korat, Reference Korat2010; Bailly & Barbour, Reference Bailly and Barbour2011), word-level and phoneme-level synchronization can be beneficial for developing reading skills, facilitating sound-written verification, promoting word recognition and assisting word-boundary detection. While the effectiveness of this method in L2 listening development has been little investigated, most of the aforementioned beneficial aspects are found to be equally advantageous for effective listening (Vandergrift, Reference Vandergrift2007). The method can consequently be used for L2 listening development to exploit its pedagogical effectiveness. However, caution is needed as the method may promote word-by-word decoding strategies, which hinders effective listening (Vandergrift, Reference Vandergrift2004). Accordingly, some complementary method may be needed to offset this effect.
2.4 Limitations of captioning and room for improvement
As mentioned earlier, the majority of studies report positive effects of full captioning, but some argue that it cannot be concluded whether the learners’ performance on the subsequent comprehension tests are based on reading the text or listening to the audio (Pujolà, Reference Pujolà2002). Winke, Gass and Sydorenko (Reference Winke, Gass and Sydorenko2013) investigated what learners usually do when they watch a captioned video using eyetracker technology. Their data revealed that when the caption was on the screen, learners read the text on average 68% of the time.
We do not clearly know how learners balance their attention between simultaneous sources of input – audio, video, and text. However, there is scepticism that the full-captioning method may promote reading over listening (Guillory, Reference Guillory1998; King, Reference King2002). Table 1 summarizes this notion and compares different captioning methods. As Table 1 suggests, word-level synchronized captioning, while fostering text-to-speech mapping, is also likely to encourage reading the captions and thus promote word-by-word decoding. Keyword captioning, on the other hand, aims to solve this problem by providing limited keywords. However, it suffers from the abrupt appearance of keywords on the screen, which makes it distractive (Montero Perez et al., Reference Montero Perez, Peters and Desmet2014a). Another major drawback of this method lies in the manual selection of keywords, which is fairly content-specific and does not consider the proficiency level of the learners, hence may not provide each learner with an adequate amount of support. In this regard, Guillory (Reference Guillory1998) concluded that the limited number of keywords in her study might not have provided enough information for the beginners involved. To manage this problem, the automatic selection of words (partial captioning) considers a more controlled selection – more personal and adapted to the learners’ needs than teacher-controlled selection. Yet it should also be noted that the video materials still have to match the learners’ proficiency level (Montero Perez et al., Reference Montero Perez, Van den Noortgate and Desmet2013).
According to Krashen’s (Reference Krashen1985) input hypothesis, learning occurs if the learner receives comprehensible input (i+1) which is slightly above his or her knowledge (i). By its very nature, authentic input is challenging as it usually contains information far beyond the knowledge of the learner (i+n). Given this fact, comprehension of such materials without any assistance leads to frustration for many L2 learners. Therefore, captioning should aim to make the authentic input comprehensible ([i+1] instead of [i+n]) for different learners.
Incentivized by this requirement, we investigate a captioning method that counters the shortcomings of current methods and allows the learners to practise for a stage where they are able to cope with authentic materials without any assistance.
3 Proposed method of PSC
Based on the discussion in the previous section, we propose partial and synchronized captioning (henceforth PSC) as an alternative to existing methods for training L2 listening comprehension. Figure 2 provides a screenshot of PSC. In this method, the original transcript is automatically reduced to a partial caption, which includes only a selected set of words or phrases (partialization) while each word is forced to appear on the screen in sync with the utterance and in a one-by-one sequence (word-level synchronization). The word-level synchronization is realized by ASR technology, which precisely maps each word to its corresponding speech segment. The filtering process of words to be presented takes into account not only the hindering factors of comprehension, but also the assessed knowledge of the learner.Footnote
Screenshot of PSC on a TED talk2
As Table 1 shows, PSC benefits from features of both partialization and synchronization. The integrated method involves the merits of each while their complementary nature counteracts the demerits of the other. For instance, word-level synchronized captioning neatly presents the word boundaries and fosters word recognition, but may promote word-by-word decoding. To address this issue, partial captioning builds on synchronized captions to provide learners with reduced transcriptions of the videos and avoid over-reliance on reading. At the same time, the word-level synchronization prevents the irregular appearance of the words in the partial or keyword captioning method. Another advantage of PSC lies in its automation, which enables the generation of appropriate captions for different learners. As mentioned earlier, compared to keyword captioning, the selection criteria in PSC are both learner-specific and content-specific (cf. infra). This is especially important because not every learner can benefit from a caption in which keywords are selected generically.
In summary, PSC aims to assist the learners to cope with spoken input difficulties without constantly referring to the verbatim caption. It acts as an intermediary stage before the learner is totally independent of the captions.
4 Selection criteria for PSC
In the partial captioning procedure, the system selects a subset of words from the transcript and presents them as captions while masking the remaining words. The effectiveness of PSC depends heavily on the choice of words to appear in the caption. Unlike keyword captioning, in PSC the major obstacles to listening comprehension are considered as a useful choice for selection.
There are different factors that may interfere with the listening comprehension process, from lexical to acoustic level. Lexical-level factors involve word frequency, specificity, idiomaticity, length, part of speech, etc. (Nissan, DeVincenzi & Tang, Reference Nissan, DeVincenzi and Tang1996; Bloomfield, Wayland, Rhoades, Blodgett, Linck & Ross, Reference Bloomfield, Wayland, Rhoades, Blodgett, Linck and Ross2010; Révész & Brunfaut, Reference Révész and Brunfaut2013), whereas acoustic-level factors are categorized by speech rate, pronunciation, accent, noise and distortion (Griffiths, Reference Griffiths1992; Buck, Reference Buck2001; Osada, Reference Osada2004; Bloomfield et al., Reference Bloomfield, Wayland, Rhoades, Blodgett, Linck and Ross2010). Moreover, content attributes such as the topic of the material and the length of the input are other influential factors (Bloomfield et al., Reference Bloomfield, Wayland, Rhoades, Blodgett, Linck and Ross2010).
In this study, words of “low frequency”, those delivered at a “high speech rate”, and those recognized as “academic or specific terms” form the basis of word selection. These features were chosen above others for being repeatedly mentioned as major contributing factors to impaired listening comprehension (Nissan et al., Reference Nissan, DeVincenzi and Tang1996; Schmitt & McCarthy, Reference Schmitt and McCarthy1997; Révész & Brunfaut, Reference Révész and Brunfaut2013). Another reason for choosing these criteria is that factors such as the speaker’s accent, noise, and the length and topic of material can be circumvented by content selection, as is also the case in this study. For instance, our videos included TED talks delivered by American speakers and trimmed to short segments. The videos are thus noise-free, do not feature non-native accents, and length is controlled. However, word specificity is considered because the videos involve technical terms. Moreover, the factors adopted in this study can feasibly be implemented and quantified automatically using existing technologies.
4.1 Speech rate
Rost (Reference Rost2005: 506) claims that speech rate is “a major factor in the comprehensibility of speech for L2 listeners”. High speech rates can negatively affect both native speakers’ and L2 listeners’ comprehension (Buck, Reference Buck2001). However, it is difficult to define a turning point where speech rate is beyond the learner’s tolerance. Nevertheless, language learners should be able to deal with normal speech rates. While defining a reliable threshold for normal rates is yet another issue, studies have reported a range between 160 to 190 words per minute (Pimsleur, Hancock & Furey, Reference Pimsleur, Hancock and Furey1977) and 3.83 to 4.66 syllables per second (Tauroza & Allison, Reference Tauroza and Allison1990).
Speech rate is often measured in words per minute, syllables per second (SPS) or phonemes per second. Of these, SPS was used in this study as it has a fairly uniform distribution and is robust against variations in speech (Wang & Narayanan, Reference Wang and Narayanan2005). Calculation of syllables is based on the structural syllabification of the text, which is realized by using the Natural Language Toolkit (Bird, Klein & Loper, Reference Bird, Klein and Loper2009). The duration of each word is measured by using the timestamps obtained from the alignment process of the ASR system. The speech rate is divided into several bins (from slow to fast) based on the standard rates of speech reported in Tauroza and Allison (Reference Tauroza and Allison1990), set as the system’s default thresholds; these thresholds can be modified according to a learner’s preference.
4.2 Word frequency
The occurrence of difficult or less frequent words is another problem in L2 listening, which may stimulate learners to pay too much attention to those words, thus hindering their listening comprehension (Goh, Reference Goh2000; Bloomfield et al., Reference Bloomfield, Wayland, Rhoades, Blodgett, Linck and Ross2010; Webb, Reference Webb2010). The term word frequency can be loosely defined as the relative number of times a given word is used in a language. In this view, the frequency of words in written or spoken corpora is related to word difficulty mainly because learners are less likely to be familiar with infrequent words (Nissan et al., Reference Nissan, DeVincenzi and Tang1996). Moreover, for both L1 and L2 listeners, processing low-frequency words takes more time (Buck, Reference Buck2001; Bloomfield et al., Reference Bloomfield, Wayland, Rhoades, Blodgett, Linck and Ross2010). Generally when watching captioned videos, good readers know how to scan for selected words (Guillory, Reference Guillory1998). Eye-tracking studies also suggest that high-frequency words are skipped more often (Rayner, 1998, as cited in Moran, Reference Moran2010) and fixated less (Inhoff & Rayner, Reference Inhoff and Rayner1986; Moran, Reference Moran2010). Partial captions should then suffice for many listeners by presenting infrequent words that otherwise interfere with comprehension.
The frequency of the word is calculated based on its occurrence in spoken or written corpora. Nation (Reference Nation2006) categorizes English vocabulary into high frequency (the most frequent 2,000–3,000 word families), mid frequency (3,000–9,000 word families), and low frequency (beyond the 9,000 frequency band). Based on this, Nation and Webb (Reference Nation and Webb2011) designed 25 word-family lists, each including 1,000 word families, plus four additional lists: (a) an ever-growing list of proper nouns; (b) a list of marginal words, including swear words and exclamations; (c) a list of transparent compounds; and (d) a list of abbreviations. The first two lists are carefully hand-selected while the rest are based on the following two famous corpora: the British National Corpus (BNC) and the Corpus of Contemporary American English (COCA).
In this study, the frequency of each word is calculated using the word-family lists and cross-checked with COCA (Davies, Reference Chang2008) to get an exact value. The BNC includes 100+ million words, whereas COCA contains 450+ million words and is hence more comprehensive. COCA is the largest corpus of American English, updated regularly, and is the most suitable for this study, which uses TED talks delivered by American speakers.
4.3 Word specificity
Word specificity here refers to special words that are used in a particular technical domain. Examples include academic terms, jargon and terminologies. According to Goh (Reference Goh2000), limited vocabulary, especially of academic words, is often a cause of problems in L2 listening comprehension. Révész and Brunfaut (Reference Révész and Brunfaut2013) noted that a higher frequency of academic words is associated with greater listening difficulty. Webb (Reference Webb2010) also showed that glossaries consisting of low-frequency word families and technical vocabularies have great value in assisting comprehension. Since the TED talks used in this study involve many academic or specific terms, word specificity is also considered as a feature for generating PSC. The system detects these academic terms by using the Academic Word List (Coxhead, Reference Coxhead2000); the Academic Vocabulary List based on COCA (Gardener & Davies, Reference Gardner and Davies2013) is also consulted to achieve higher accuracy.
Finally, we designed and implemented a system to generate PSC in order to provide adequate support for L2 listeners. Figure 3 depicts the process flow and the main components of this system. Detailed information on the implementation of this system can be found in Appendix I.
System architecture and process flow
Given the novelty of the PSC method, we conducted an experiment to investigate the following research questions about the potential effectiveness of PSC:
-
1. Do captioned videos (using PSC or full captions) lead to better comprehension than non-captioned videos?
-
2. Can PSC be substituted for the conventional full-text captioning method?
-
3. Do proficiency differences affect the usefulness of PSC?
-
4. Does PSC help the learner comprehend a subsequent segment of the video without any captions?
5 Methodology
5.1 Participants
Participants were 58 Japanese students in two classes (28 in class A, 30 in class B) enrolled in computer-assisted learning (CALL) courses at Kyoto University. The subjects were 19- to 22-year-old engineering students, most of whom had started studying English from the age of 10 to 13. The participants’ scores on the CASEC (Computer Assessment System for English Communication)Footnote 3 test ranged from 560 to 850, indicating that they had different proficiency levels. This is important for the design of the study to investigate the effectiveness of PSC for different proficiency levels. Participants were divided into three groups based on their CASEC scores: beginners (560–599), pre-intermediates (600–759) and intermediates (760–850). CASEC is a standard test which evaluates the learners on their knowledge of vocabulary and listening ability with approximately 0.96 reliability (Nogami & Hayashi, Reference Nogami and Hayashi2010).
5.2 Materials
5.2.1 Video selection
The clips were selected from the TED website which provides authentic videos plus almost accurate transcripts under the Creative Commons license.
The selection was carefully made to include videos of American speakers only, to avoid the influence of other accents. Moreover, caution was taken to exclude videos that contained many difficult technical terms. The average speech rate and word frequency of the videos were calculated to exclude those that were too fast or included too many infrequent words. As Figure 4 shows, the nine videos selected shared roughly comparable speech rates and word frequencies. All videos were trimmed to approximately five-minute meaningful segments.
Statistics on the speech rate and word frequency of the video clips
5.2.2 Caption preparation
Each video was prepared with three settings of captions: no caption (NC), full caption (FC) and PSC. PSCs were generated for different proficiency groups based on the vocabulary size and the tolerable rate of speech (see Section 5.3). The percentage of words shown in the final captions, however, did not exceed 30% of the transcript across each proficiency level, for any of the videos. Figure 5 shows statistics for the PSC generated for pre-intermediates. As the figure illustrates, the percentage of words shown is different, based on each acting feature (i.e. speech rate, frequency and specificity). In the experiment, only the PSCs generated based on all three features were used.
Percentage of words shown in PSC for pre-intermediate group
5.3 Data collection instruments
5.3.1 Vocabulary size test (frequency thresholding)
A vocabulary size test designed by Nation and Beglar (Reference Nation and Beglar2007) with an approximate Rasch reliability of 0.96 was used to evaluate the participants’ vocabulary reservoir. The results of this online test were used to determine the frequency threshold for our caption generator. This test consists of 140 multiple-choice questions, with 10 items from each 1,000-word family. Since the caption generator uses the same 25 word-family lists as its references (cf. Section 4.2), the result of the test is appropriate as a threshold.
5.3.2 Speech rate test (speech rate thresholding)
The subjects were given several short conversations taken from a TOEIC® practice test with approximate reliability of 0.90, as reported by ETS.Footnote 4 The speed of the conversations was modified into three different levels: slow, moderate and fast. The participants were asked to answer some questions about the related conversation and report whether the audio was too slow, too fast or appropriate for them. Data on this test was only used to define suitable speech-rate thresholds for the system, mapped to the standard rate of speech (cf. Section 4.1), in order to adjust PSC for each group; the scores were not counted in the analysis.
5.3.3 Comprehension tests
Once the participants had watched a video with a specific caption type, they were asked to take a comprehension test in the form of multiple-choice and cloze questions. The multiple-choice questions focused both on the main idea and on the specific information in the video. In the cloze test, the participants listened once to a short audio file (20–30 seconds) extracted from the video they had watched and filled in the blanks in a corresponding paragraph (verbatim text of the audio). The missing words were selected from those that had appeared in the caption in order to constrain the choice of words and differentiate between the conditions. Appendix II includes a sample of the questions.
5.3.4 Questionnaires
Since the proposed PSC was being used for the first time in a real language learning environment, we conducted a 5-point Likert-scale questionnaire to solicit feedback on the PSC system and to identify opportunities for its future enhancement. The questionnaire consisted of twelve items: two questions focusing on the participants’ attitudes toward using PSC, four questions comparing their usage of PSC and FC, two questions focusing on the synchronization feature, two questions investigating the partialization aspect and the selected words in PSC, and the last two questions on the readability of the captions. The questions were reviewed by several experts to assure face and content validity. Appendix III includes the questionnaire and the participants’ feedback.
5.4 Procedure
The experiment was conducted once a week in CALL classes and lasted for four sessions. Each experimental session took approximately 60 minutes. Figure 6 illustrates the experimental procedure. The first session was dedicated to the vocabulary size test and the speech rate test in order to generate appropriate PSC for each video (adjusted for each proficiency group). In this session the students in class B had a trial experiment with PSC.
Experimental procedure
The experiment was conducted in two parts. Part I evaluated the effectiveness of each captioning method (NC, FC, PSC) on learners’ comprehension and provided data for research questions 1–3 by allowing a comparison among these three methods and evaluating PSC for different proficiency levels. Part II was designed to evaluate the immediate effect of each captioning method on preparing the participants to listen without any captions. This part relates to research question 4 to investigate if PSC can help the learner comprehend the video later without any captions.
To realize this design, each video was divided into two segments: 70% from the beginning of the clip and the subsequent 30% from the last cropped frame to the end of the clip. The longer part of the clip was used for Part I of the experiment and hence was captioned with NC, FC or PSC, while the shorter part of the clip was preserved for Part II of the experiment without captions. First, the subjects completed Part I and watched a video with NC, FC or PSC, followed by a comprehension test. Next, the subjects did Part II and watched the rest of the same video without any captions, followed by another comprehension test.
In the final session, the subjects filled out the questionnaire about the effectiveness of PSC. The same procedure was adopted in both classes to maintain similar conditions. As shown in Figure 7, the types of captions were rotated between the two classes to alleviate the effect of content variability. The order of watching was also rotated.
Experimental design for Part I (70% of videos with treatment)
6 Data analysis and results
An analysis of variance (ANOVA) test was used to examine the effect of conditions (type of caption) on comprehension, with the significance level set to 0.05. The effect size (ηp 2) is reported and interpreted based on Cohen’s rules of thumb: small (ηp 2>.01), medium (ηp 2>.06) and large (ηp 2>.14). Fisher’s least significant difference (LSD) post-hoc test was used to compare the effect of captioning methods. Finally, a paired-sample t-test was used to check the differences between PSC and FC conditions across three proficiency groups.
6.1 Research question 1
The first research question investigated whether the use of captions (FC or PSC) can lead to better comprehension compared to the no-caption condition (NC). This question deals with Part I of the experiment in which the participants watched 70% of videos with different captioning methods (NC, FC, PSC). Table 2 summarizes the mean scores on this part. The results presented in Table 3 reveal that scores under FC and PSC conditions were statistically higher than the NC condition. Therefore, the caption type significantly affected comprehension scores [F (1, 57)=59.5, p<.05, ηp 2 =.51].
Mean scores and standard deviations on comprehension tests – Part I
Repeated-measure ANOVA on the effect of three types of caption – Part I
Fisher’s LSD post-hoc analysis (Table 4) revealed that the participants’ scores under the PSC condition (M=52.89, SD=19.39) and the FC condition (M=54.25, SD=17.33) were significantly higher than the NC condition (M=35.69, SD=14.68). This finding provides a positive answer to the first research question.
LSD post-hoc comparisons of scores gained under different conditions – Part I
6.2 Research question 2
The second research question asks if PSC can be substituted for FC, and relates to Part I of the experiment. As Table 4 shows, the difference between the scores of the PSC and FC conditions was not statistically significant [F (1, 57)=.25, p=.62, ηp 2=.004]. The findings suggest that PSC, while presenting less than 30% of the text, leads to a statistically equivalent level of comprehension as FC, which presents 100% of the transcript.
6.3 Research question 3
The third research question relates to Part I of the experiment and concerns the effectiveness of PSC for different proficiency groups. Results of the paired-sample t-test in Table 5 revealed that within each proficiency group, the average scores of the members under the FC condition and the PSC condition are not significantly different: beginners (t (18)=.22; p=.83), pre-intermediates (t (18)=−1.09; p=.23) and intermediates (t (19)=.23; p=.98). While three levels of PSC were used with different amounts of text for each proficiency group, the finding indicates that PSC successfully adjusted its content to the learners’ proficiency levels for a similar level of comprehension with FC.
Paired-sample t-test on scores of PSC vs. FC conditions across proficiencies – Part I
6.4 Research question 4
This question investigates the effectiveness of PSC on preparing learners for real-life contexts and concerns Part II of the experiment – i.e. watching the subsequent 30% of videos without any captions after the previous segment (70%) had been seen with captions. As shown in Table 6, the best scores were gained when the learners first watched videos with PSC (M=56.59, SD=17.34) compared to NC (M=40.12, SD=17.39) or FC (M=42.65, SD=13.37) conditions.
Mean scores and standard deviations on comprehension test – Part II
Table 7 shows that this difference is statistically significant based on Fisher’s LSD test. This means that the learners’ comprehension of an uncaptioned video was significantly higher when they had watched a previous segment with PSC rather than with FCs or NCs. [F (2,118)=20.5, p <.05, ηp 2=.26]. Although this is a short-term enhancement (partly because of adaptation to the video), this finding is still promising.
LSD post-hoc comparisons of scores gained under different conditions – Part II
We also compared the scores of participants when they watched the videos with no captions in Part I (M=35.69, SD=14.68) and the remainder of the same videos without captions in Part II (M=40.12, SD=17.39). Under these two similar conditions, there was no statistically significant difference between the scores [t (57)=−1.296; p=.20]. Consequently, we can be confident that the results on Part II of our experiment are affected by the treatment (caption) used in Part I.
7 Discussion
7.1 Overall effect of different captioning methods
The first research question aimed to compare the effect of two captioning conditions (FC or PSC) with an NC. Our quantitative results on this question corroborate the findings of previous studies and suggest that the presence of captions significantly aids listening comprehension (Markham & Peter, Reference Markham and Peter2003; Danan, Reference Danan2004; Taylor, Reference Taylor2005; Winke et al., Reference Winke, Gass and Sydorenko2010; Montero Perez et al., Reference Montero Perez, Van den Noortgate and Desmet2013). This is confirmed regardless of whether PSC or FC was used, and is in line with Paivio’s dual coding theory (see Section 2.1). When reading captions is part of watching a video, learners can benefit from multiple input modalities. In this regard, Bird and Williams (Reference Bird and Williams2002) emphasized that the use of text and sound results in better recognition memory.
However, even in the FC condition, the participants’ scores on the tests are below 60%, which indicates the difficulty in understanding the video content. TED talks are apparently difficult to comprehend for most of our non-native speakers and are more appropriate for advanced learners whereas the participants in this study were beginner to intermediate level.
7.2 Effectiveness of PSC compared to FC
In the context of research question 2 – comparing the effectiveness of PSC with FC – we found that the test scores under FC and PSC condition were not statistically different. This may indicate that the two methods can be used interchangeably. However, PSC, by presenting fewer numbers of words (less than 30%), encourages more listening than reading, compared to FC, which has been criticized for promoting reading over listening (Pujolà, Reference Pujolà2002; King, Reference King2002). But how can PSC be as effective as FC for comprehension while it only shows a small amount of the text? The following two reasons are considered.
-
1. Cognitive load reduction. While PSC improves comprehension following the assumption of dual coding theory, the effectiveness of this method may also be explained by the cognitive load phenomenon and limitation of working memory that occur in cases of excessive information (Baddeley, Reference Baddeley1992; Sweller, Reference Sweller1994). With smaller amounts of text in the visual channel, learners are less likely to encounter overload to multichannel processing and more likely to achieve fuller comprehension of the information coming through the auditory channel (Guillory, Reference Guillory1998: 97). Thus, for effective learning, multimedia instruction should minimize any unnecessary cognitive load (Mayer & Moreno, Reference Mayer and Moreno2003). In this view, PSC aims to foster listening by providing minimal assistance and hence acting as a trade-off between dual coding theory and cognitive overload. Moreover, its word-level synchronization facilitates text-tracking and reduces the amount of scanning for text-to-speech mapping.
-
2. Appropriate selection criteria. A further explanation to strike a balance between these theories (dual coding and cognitive load) may lie in the appropriate selection criteria adopted by PSC: with appropriate selection of caption text, learners can gain better comprehension without excessive cognitive load. As such, the results of this study are mainly based on the three features: speech rate, word frequency and specificity, which underlie PSC and form its selection criteria. Rationale of these selection criteria can be found in Griffiths (Reference Griffiths1992), Zhao (Reference Zhao1997) and Buck (Reference Buck2001) on speech rate, and in Nissan et al. (Reference Nissan, DeVincenzi and Tang1996), Webb (Reference Webb2010) and Révész & Brunfaut (Reference Révész and Brunfaut2013) on frequency and specificity of words.
7.3 Effectiveness of PSC across proficiency levels
The results revealed that the scores gained under the FC and PSC conditions were not statistically different within each proficiency group. In other words, the subjects in each proficiency group could gain similar levels of comprehension under the PSC condition (with less than 30% of the text shown) and the FC condition (where 100% of the text is shown). PSC tries to provide enough assistance by adapting the amount of text in the caption to the learners’ needs and proficiency levels.
Other studies that have explored the use of captions for learners with different proficiency levels have reported mixed results. The results in this study are in line with Markham’s (Reference Markham1989) and Guillory’s (Reference Guillory1998) findings on the effectiveness of captions (full and keyword respectively) for low-proficiency learners and therefore contradicts Taylor (Reference Taylor2005), who reported that first-year students scored better in an NC than an FC condition. A possible explanation may lie in the noteworthy feature of PSC that evaluates the learner’s level (i), adjusts the amount of textual information to match that level and hence provides comprehensible input by changing i+n to i+1 for different learners. However, due to the limited number of participants in each group, the results must be interpreted with caution and cannot be extrapolated.
7.4 Effectiveness of PSC to prepare learners for real-life situations
The results of our final research question confirmed that after watching videos with PSC, the participants performed better in comprehending a video without any assistance.
Reliance on captions is an individual matter that cannot be universally applied. Consequently, before considering the addition or removal of captions, instructors should have evidence to the degree at which the individual learners rely on captioning support for comprehension (Leveridge & Yang, Reference Leveridge and Yang2013: 211).
As explained earlier, PSC can provide different amounts of assistance to learners at different levels. More importantly, it allows the learner to adjust the quantity of textual clues by changing the thresholds. This method acts as a source of scaffolding and allows the learners to prepare for the NC condition at their own pace. It is anticipated that following this strategy over the long term, learners will be able to rely entirely on their listening skills for comprehension. Given the nature of listening skills, however, a long-term experiment is required to confirm this effect.
One limitation in this study concerns Part II of the experiment on the immediate post-effect of captioning methods, where the results may reflect only a temporary improvement and should be confirmed by further research. Moreover, our participants were beginner to intermediate level Japanese learners of English, so the results cannot be generalized to other participants with different L1 or different proficiency levels. Administering a longitudinal study and involving more participants would form a more solid basis for analysis.
8 Conclusions
The study introduced a smart type of captions that allows the use of limited textual clues and promotes listening to the audio in order to comprehend the material. The proposed method, PSC, is based upon three factors that contribute to listening difficulty: speech rate, word frequency and specificity. Using these features, the system generates a caption that tries to deal with the limitations of previous methods. With the ASR technology, the system emulates the speech flow, which facilitates text-to-speech mapping and avoids the salient appearance of the words on the screen. The system assesses the tolerable speech rate and vocabulary size of the learner and adjusts the caption to his or her proficiency level.
Evaluated in two CALL classes, the results of the experiments showed that learners scored better when using PSC compared to the NC condition, and in comparable comprehension to the full-caption condition. Furthermore, the learners gained significantly higher scores on a new segment of the video without any captions when they had previously watched the first part of the video with PSC. The finding highlights the positive effect of PSC on preparing learners for listening in simulated real-life situations, where they do not have any means of assistance such as captions or speed controllers. The results also indicate that the method can assist learners in adequate comprehension of videos by presenting less than 30% of the transcript. This method is expected to be effective particularly for Japanese students who rely heavily on caption text in order to comprehend the content of a video.
We conclude this paper with some suggestions for future directions. We need to enhance this system to encompass other features that affect L2 listening comprehension. This is crucial for increasing the accuracy of word selection in PSC and hence providing better assistance to learners. A wide range of features such as accent, co-articulation, pragmatics, idiomaticity, part of speech and even the cultural implications of words could be considered. In addition, the system should constantly analyse the user’s improvement over time so that the captions can be adapted to suit individual learners. To this end, and to help teachers monitor their students’ progress, a history of the learner’s performance should also be aggregated within the system. Furthermore, the current version of PSC, which supports English, might be extended to other languages for L2 learners to enable PSC to be used as an alternative training tool for L2 listening development.
Supplementary materials
For supplementary material/s referred to in this article, please visit https://doi.org/10.1017/S0958344017000039
