RESPEAKING AS A HYBRID MODALITY

Despite being a newcomer on the AVT scene, respeaking seems to have some “elder siblings” in the translation/interpreting family. First of all, respeaking (especially its interlingual variety) can be likened to media interpreting on television, where the interpreter interprets live televised content for the TV audience (Kurz, Reference Kurz, Garzone and Viezzi2002; Pignataro, Reference Pignataro2011; Pöchhacker, Reference Pöchhacker, Gambier and van Doorslaer2010). Both respeakers and media interpreters are bound by strict time constraints since they are often instructed to keep as close to the original speaker as possible (Pignataro, Reference Pignataro2011) and to translate at a “supersonic pace” (Bros-Brann, Reference Bros-Brann1994, p. 26), which undoubtedly affects their EVS. Similarly to media interpreting on TV, long EVS is not desirable in respeaking as it increases the delay in displaying subtitles on screen. Second, the media interpreter addresses two types of audiences at the same time: on-screen participants (speakers in the TV studio such as a talk-show host and guests) and off-screen participants (TV viewers; Sergio, Reference Sergio and Chapelle2013). In the same vein, respeakers deal with the televised dialogue on the one hand, and with their deaf and hard of hearing audience at home, on the other. Sergio (Reference Sergio and Chapelle2013) noted that it is possible for media interpreting and subtitling to coexist in the same TV program: when the interpreter's words are subtitled by another person. We now know that these two roles can be performed by a single person: an interlingual respeaker. Sergio (Reference Sergio and Chapelle2013) claims that media interpreting “requires additional skills and a new professional profile” (p. 2) and cites Bros-Brann (1993, p. 1), stating that it requires “an entirely new mindset compared to everyday practice of conference interpretation and to what all of us have learned and taught in various schools of interpretation.” We believe the same applies to respeaking as it now calls for new research and training.

Respeaking also shares some ground with simultaneous film interpreting (Russo, Reference Russo2005), found at film festivals and other live events. The common ground lies in a complex audiovisual situation, combining inputs from the visual, oral, and written codes. Among the similarities shared by simultaneous film interpreting, subtitling, and respeaking are also the need to synchronize the translation with the image and to reduce the original text, as well as the fact of being “caught in the relationship between the written and the oral” (Gambier, Reference Gambier2003, p. 178).

Finally, respeaking has often been likened to simultaneous conference interpreting (Eugeni, Reference Eugeni2008b; Marsh, Reference Marsh2004; Romero-Fresco, Reference Romero-Fresco2011). Both interpreting and respeaking demand excellent multitasking and split attention skills, as they require listening to the original text and rendering it into the same or another language, simultaneously monitoring the output (Jones, Reference Jones2002; Pöchhacker, Reference Pöchhacker2004; Romero-Fresco, Reference Romero-Fresco2011). Unlike interpreters, however, respeakers also need to add the necessary punctuation marks, and condense the original text to fit the limited space of subtitles on the screen. One of the most important similarities in the process of respeaking and interpreting is what is known as the EVS.

EVS

Similarly to respeaking, simultaneous interpreting involves concurrent management of two speech channels: listening to the source language as well as producing and monitoring the target language. In order to do that, interpreters may to a certain extent produce the target text during pauses in the source text (Barik, Reference Barik1973; Goldman-Eisler, Reference Goldman-Eisler1972; Goldman-Eisler, Dechert, & Raupach, Reference Goldman-Eisler, Dechert and Raupach1980), but for most of the time (estimated at 70% by Chernov, Reference Chernov, Lambert and Moser-Mercer1994), they listen and speak at the same time. Thus, the time lag or delay between the moment the original utterance is spoken out and the moment when the interpreter produces his/her equivalent in the target language, known as the EVS or décalage, is one of the most important features of processing involved in simultaneous interpreting (Lee, Reference Lee2002; Pöchhacker, Reference Pöchhacker2004). According to Timarová, Dragsted, and Hansen (Reference Timarová, Dragsted, Hansen, Alvstad, Hild and Tiselius2011), EVS “provides insight into the temporal characteristics of simultaneity in interpreting, speed of translation and also into the cognitive load and cognitive processing” involved in simultaneous interpreting (p. 121), and it is a reliable and quantifiable measure of cognitive processing (Lee, Reference Lee2002). It is also an important skill to be practiced in simultaneous interpreter training (Bartłomiejczyk, Reference Bartłomiejczyk, Chmiel and Janikowski2015).

A number of previous studies have attempted to establish a minimum or optimum EVS unit. EVS can be measured in units of time (e.g., seconds) and/or linguistic units, such as a number of words or the nature of syntactic phrases. Paneth (Reference Paneth1957) measured EVS values and found they were between 2 and 4 s, a result which was later confirmed by Barik (Reference Barik1973) and Oléron and Nanpon (Reference Oléron, Nanpon, Pöchhacker and Shlesinger2002/1965). Lederer (Reference Lederer, Gerver and Sinaiko1978) found that EVS fell between 3 and 6 s. Other scholars reported the average EVS amounting to 2 s (Christoffels & de Groot, Reference Christoffels and de Groot2004), 2.68 s (Defrancq, Reference Defrancq2015), and 4.7 s (Timarová et al., Reference Timarová, Dragsted, Hansen, Alvstad, Hild and Tiselius2011). Schweda-Nicholson (Reference Schweda-Nicholson1987) found EVS to range from 5 to 10 words or several seconds, while Gerver (Reference Gerver1969) argued that the average delay at average presentation rates is about 4–5 words. Goldman-Eisler (Reference Goldman-Eisler1972) suggested that EVS units depend more on the syntax than on the lexis, and established the minimum EVS unit to be a predicative expression (noun phrase and a verb phrase). Adamowicz (Reference Adamowicz1989) also measured EVS in terms of meaningful syntactic units and nominal/verbal phrases as frequent EVS units in English–Polish interpreting. Donato (Reference Donato2003) applied an extended typology of syntax-based EVS units and found that the most frequently applied EVS units consisted of a noun phrase and a verb phrase. Christoffels and de Groot (Reference Christoffels and de Groot2004) noticed a certain consistency pertaining to the average EVS duration reported in various studies with varied methodologies. They claimed that the upper boundary of the EVS is related to memory capacity (longer EVS increases memory load), while the lower boundary is related to the minimum meaningful unit (Lederer, Reference Lederer1981; Setton, Reference Setton1999) needed for the interpreter to decode meaning and perform interpreting.

EVS has also been linked to other factors, such as source text delivery rate (Lee, Reference Lee2002), working with or without text (Lamberger-Felber, Reference Lamberger-Felber2001), and sentence length (Lee, Reference Lee2002). Barik (Reference Barik1973) measured temporal characteristics of interpretation of four types of texts: spontaneous speech, semiprepared material, prepared “oral” material (a speech written to be read out), and prepared “written” material (an article). He found that interpreted texts usually included speaking for a greater proportion of the time than the original texts, and this proportionality was greater for scripted than unscripted texts, as the former have higher information density. However, due to a low number of participants, Barik did not find a consistent pattern of results when it comes to text types. Timarová et al. (Reference Timarová, Dragsted, Hansen, Alvstad, Hild and Tiselius2011) found a significantly smaller EVS when interpreting figures as opposed to verbs or beginnings of sentences, which means that interpreters try to produce figures as quickly as possible in order not to burden the memory with such difficult noncontextual items with high informative content (Chmiel, Reference Chmiel, Chmiel and Janikowski2015; Mazza, Reference Mazza2001). Adamowicz (Reference Adamowicz1989) found smaller EVS in interpreting a spontaneous text as compared to prepared texts, while Díaz-Galaz, Padilla, and Bajo (Reference Díaz-Galaz, Padilla and Bajo2015) reported smaller EVS following advance preparation for the simultaneous interpreting task. It also seems that more experienced interpreters work with smaller EVS, as Timarová, Čeňková, and Meylaerts (Reference Timarová, Čeňková, Meylaerts, Ferreira and Schwieter2015) found a negative correlation between median EVS and days of interpreting experience. Taken together, these studies suggest that EVS depends on a combination of global (such as language combination) and local (such as propositions in the text) factors.

A generally held view is that longer EVS is better than keeping very close to the speaker (Kade, Reference Kade1967). Interpreting trainees are taught to prolong their EVS in order to avoid word-for-word interpreting and in order to have enough time to restructure and reformulate the message and to express it naturally in the target language (Bartłomiejczyk, Reference Bartłomiejczyk, Chmiel and Janikowski2015; Chmiel, Reference Chmiel, Chmiel and Janikowski2015; Gorszczyńska, Reference Gorszczyńska, Chmiel and Janikowski2015). Shorter EVS is advised when information density in the text increases (e.g., due to enumerations or numerical data) because it lowers the memory load and leads to fewer omissions. Empirical studies have shown that shorter EVS does not have to lead to poorer quality (Defrancq, Reference Defrancq2015) or even may be associated with better quality and higher accuracy (Lee, Reference Lee2002; Timarová et al., Reference Timarová, Čeňková, Meylaerts, Hertog, Szmalec and Duyck2014). It seems that the interpreter has to strike a balance between the EVS that is not too short (so that meaning can be constructed and restructuring is possible) and not too long (so that there is no memory overload leading to lower accuracy of interpretation; Christoffels, de Groot, & Kroll, Reference Christoffels, de Groot and Kroll2006; Kade & Cartellieri, Reference Kade and Cartellieri1971). In a sense, EVS lengthening and shortening can be the interpreter's strategic choice and is listed as one of the tactics to be used by interpreters to prevent the occurrence of problems (Gile, Reference Gile2009).

As opposed to interpreting, where longer EVS is usually not problematic for the target audience, in respeaking, “décalage is less desirable” (Romero-Fresco, Reference Romero-Fresco2011, p. 107) because it causes delay in the live subtitling production process and effectively increases the gap between the on-screen images and the subtitles accompanying them. Such delay, as mentioned above, is a cause of great distraction to deaf and hard of hearing viewers and makes a TV program with live subtitles difficult to follow as the subtitles simply cannot catch up with the images (Mikul, Reference Mikul2014).

PAUSES

Important as it is, EVS is not the only time-related factor that is crucial in respeaking. Another significant factor are pauses made by respeakers in their speech. In its traditional sense, pauses “serve to divide discourse into tone groups and organize it into information units” (Shlesinger, Reference Shlesinger, Lambert and Moser-Mercer1994, p. 229); some pauses, however, are a result of hesitations (Pignataro, Reference Pignataro2011, p. 87). We can therefore see that some pauses are more desirable than others that are unwanted and can be considered “linguistically detectable faults” (Goffman, Reference Goffman1981, p. 172) and “disfluencies” (Garnham, Reference Garnham1985, p. 206). The undesirable pauses may be an indication of increased cognitive effort on the part of the speaker, or, as Mead (Reference Mead2000) put it, “manifestations of the effort of reasoning and formulation which accompany linguistic production” (p. 91). Skilled professional speakers are able to control their linguistic output, minimizing the unwanted pauses in production that may “betray moments of doubt or distraction” (Mead, Reference Mead2000, p. 91); see also Goffman (Reference Goffman1981).

In interpreting, pauses are an important element of fluency, which is thought to be a distinguishing feature between a professional and trainee interpreter (Mead, Reference Mead2000). In respeaking, pauses are used as an important element of interaction between the respeaker and the SR software. In order to work properly, SR requires a short pause (less than 1 s) to be made by respeakers before it will transfer speech into text (Jurafsky & Martin, Reference Jurafsky and Martin2008; Romero-Fresco, Reference Romero-Fresco2011). Dictating one word at a time is not a good option, as it would remove the language context from the SR process, which is responsible for its high word accuracy (Romero-Fresco, Reference Romero-Fresco2011). Therefore, respeakers need to stop dictating after a few words and make a pause in appropriate moments in order for the SR program to turn the spoken words into text that can then be displayed as subtitles. It needs to be noted here that subtitling is a constrained medium in itself: there is a limit to the number of lines and to the number of characters per line that can be used so that people can both read the subtitles and follow the on-screen images. Given all the above, respeakers are advised to divide the text they respeak into short, self-contained meaningful chunks, known as “respeaking units.” Romero-Fresco (Reference Romero-Fresco2011, p. 108) defines respeaking units as “idea units that lend themselves to accurate recognition by the SR software (phrases as opposed to single words) and to comfortable reading for the viewers (around one line in a one-, two- or three-line subtitle).” An ideal respeaking unit is believed to consist of five to seven words (Romero-Fresco, Reference Romero-Fresco2011). Longer stretches of text may cause problems with subtitle segmentation, extending over too many lines, and as such be difficult to follow by viewers.

Pauses, both in respeaking and in interpreting, may result from pauses made by the original speaker and/or the speech production process conducted by the interpreter/respeaker. In regular speech production, pauses serve a number of functions: they enable breathing, may have a semantic or rhetorical function, and “provide time to cope with difficulties which can arise at any point in the speech production cycle” (Tóth, Reference Tóth2013, p. 3). In simultaneous interpreting, silent pauses can manifest problems with source text comprehension, lexical search for the translation equivalent, and difficulties with expressing a given sense in the target (Bartłomiejczyk, Reference Bartłomiejczyk2006; Piccaluga, Nespoulous, & Harmegnies, Reference Piccaluga, Nespoulous and Harmegnies2005; Tóth, Reference Tóth2011). As Piccaluga et al. (Reference Piccaluga, Nespoulous and Harmegnies2005, p. 151) state: “it would be reasonable to think that the proliferation and/or lengthening of these pauses is linked to difficulties of various kinds in performing the complex task of interpreting.”

Cecot (Reference Cecot2001) examined silent pauses produced by professional interpreters and concluded that the most frequent were segmentation pauses (defined as grammatical and serving a communicative function) followed by hesitation pauses (defined as nongrammatical without a communicative function) and rhetorical pauses. Cecot's comparison of pauses in the source and target texts led her to conclude that interpreters generally followed the speaker's pattern of segmentation and rhetorical pauses. However, these patterns also reflect additional linguistic processing and cognitive effort (Goldman-Eisler, Reference Goldman-Eisler1972). This is why studies have shown that there are more pauses in the interpreted text as compared to the source text (Tissi, Reference Tissi2000). Tissi (Reference Tissi2000) found that interpretations included fewer but longer silent pauses than source texts, which suggests that pausing in interpreting might to a large extent stem from linguistic processing that differs from language planning in regular speech production. Pauses might be strategic (to gain time before a correction) or stem from problems in the simultaneous interpreting technique experienced by interpreting trainees (Tissi, Reference Tissi2000). Trainees have been found to pause more than professional interpreters, suggesting more cognitive effort expended by trainees for restructuring and lexical retrieval (Tóth, Reference Tóth2013).

Pausing patterns are also influenced by other factors, such as the direction of interpretation, source text speed, and sound quality. Interpreters were more fluent and paused less when working into their native language in both simultaneous (Piccaluga et al., Reference Piccaluga, Nespoulous and Harmegnies2005) and consecutive interpreting (Mead, Reference Mead2000). Piccaluga et al. (Reference Piccaluga, Nespoulous and Harmegnies2005) studied a simultaneous interpreting task with manipulated source text speed and sound quality (added noise interference). They analyzed both the number and the length of pauses. Their participants paused more when the source text was of lower quality and delivered faster.

Interpreting experience may also influence pausing patterns. In general, the more experienced the interpreter, the shorter the produced pauses. In a study by Tóth (Reference Tóth2013), pauses made by trainees lasted for 250–750 ms on average, and they were longer than the pauses made by professionals (those ranged between 200 and 500 ms). The findings by Piccaluga et al. (Reference Piccaluga, Nespoulous and Harmegnies2005) suggest that with increased interpreting experience, the participants seem to produce shorter but more numerous pauses. However, less experienced participants produced longer pauses, manifesting processing difficulties and breakdowns and leading to speech flow disruptions. The mean length of pauses in this study was 768 ms across all participants.

Taken together, the studies on silent pauses in interpreting suggest that pauses do reflect linguistic processing difficulties and serve as an index of disfluency (Pöchhacker, Reference Pöchhacker2004). Pauses tend to be longer when the source text is more difficult (delivered faster or with a lower sound quality) and when the interpreter is less experienced and works from the mother tongue into the foreign language.

To recap the discussion on EVS and pauses, respeakers need to minimize their EVS, following the original utterance quite closely, and they must make short pauses between self-contained units of meaning (“respeaking units”) in order for the SR software to work efficiently. Respeakers must therefore find “a good speech-to-pause rhythm, given that no subtitle will be shown until a pause is made” (Romero-Fresco, Reference Romero-Fresco2011, p. 108).

THE PRESENT STUDY

The goal of the present study is to better understand the process of intra- and interlingual respeaking of different TV genres, in particular in terms of temporal characteristics like EVS and pauses. By conducting this study, which, to the best of our knowledge, is the first of its kind, we also set out to investigate whether previous interpreting and translation experience may be an important factor affecting temporal characteristics of respeaking.

The profession of a respeaker is in its nascent stage, and there were no professional respeakers in Poland at the time when this study was conducted. Thus, in order to establish whether interpreters and translators would produce respoken subtitles with less delay than noninterpreting and nontranslating bilingual controls, we exposed them to intensive 2-day training in respeaking fundamentals. In the first experiment, we compared their performance in intra- and interlingual respeaking tasks. In the second experiment, we decided to probe deeper into text characteristics that could influence temporal features of respeaking in intralingual tasks only.

EXPERIMENT 1

Method

In the first experiment we wanted to see how EVS and pauses are modulated by the nature of the respeaking task: interlingual versus intralingual. In addition, we wanted to compare three groups of participants: interpreters (as those who have experience both with temporal constraints characteristic for interpreting and respeaking, and with interlingual processing), translators (as those who have experience in copying with interlingual processing in translation), and bilingual noninterpreting controls (who have experience neither with temporal constraints nor with interlingual processing). Thus, we used a mixed factorial design with task (intralingual and interlingual respeaking) as a within-subject independent variable and group (interpreters, translators, and bilingual controls) as a between-groups independent variable.

We predicted that the interlingual respeaking task will be more difficult than intralingual respeaking, which will be manifested by longer EVS and longer pauses in the interlingual condition. We also expected that interpreting and translation experience would modulate EVS and pause length in both tasks. We predicted that in the interlingual condition, interpreters would respeak with shorter EVS and would produce shorter pauses than translators and controls, because they are used to coping with demanding time constraints when interpreting and because they are regularly exposed to interlingual processing. We also expected that translators would outperform controls (i.e., manifest shorter EVS and pauses) in the interlingual, but not in the intralingual condition, because they are used to interlingual processing in their translation experience. In addition, we predicted that translators and controls would have more pauses than interpreters as compared to the pauses in the original text.

Materials

In the interlingual condition, we used a slow-paced fragment of a speech delivered by President Barack Obama on the 25th anniversary of Polish Freedom Day in June 2014 in Warsaw. In the intralingual condition, we used a slow one-speaker speech, the New Year's address by Poland's Prime Minister Ewa Kopacz.

The clips were matched in terms of the number of speakers, genre, and speech rate measured by words per minute. As shown in Table 1, although matched for duration and number of words, the clips differed in terms of the number of syllables and speech rate measured as the number of syllables per duration. This discrepancy stems from the fact that Polish words are on average longer than English words. Both clips represented prescripted speech.

Table 1.

Characteristics of video clips used in Experiment 1

Participants

In this study we tested a total of 57 participants (50 women, 7 men). They were volunteers recruited among professional interpreters and translators as well as graduates and final-year students at the Institute of Applied Linguistics at the University of Warsaw, the Faculty of English at Adam Mickiewicz University in Poznań, and the University of Social Sciences and Humanities in Warsaw as well as through social media (AVT Lab and RespeakingProject Facebook pages).

Their mean age was 27.48 (SD = 5.71), ranging from 21 to 51. Based on the self-reported experience in interpreting, participants were divided into three groups: 22 interpreters (with at least 2 years of exposure to interpreting either in a professional context or during an intensive academic program), 23 translators (with at least 2 years of exposure to translation either in a professional context or during an intensive academic program), and a control group including 12 participants with no previous experience in translation or interpreting.

None of the participants had any respeaking experience. Therefore, prior to the respeaking test, all participants underwent a 2-day (16 hr in total) intensive training in respeaking. They were trained in the fundamentals of respeaking, including linguistic and technical skills, such as management of simultaneous resources, working memory, monitoring their respoken output, enunciation, and punctuation as well as the creation of voice profiles in the Newton Technologies SR software and the FAB Subtitler Live subtitling program.

The participants were instructed to respeak in idea units or respeaking units (Romero-Fresco, Reference Romero-Fresco2011, pp. 60, 108), that is to say, whole phrases rather than individual words and to pause after the entire phrase has been uttered for the SR language and acoustic model to work effectively. An ideal respeaking unit is considered to be between five and seven words (Romero-Fresco, Reference Romero-Fresco2011); longer stretches of text may not be comfortable for viewers to read and may result in increasing the delay between the image and the accompanying subtitles.

Procedure

Participants were tested individually in a research lab. Before the test, informed consent was obtained from each participant. The respeaking test started with the experimenter familiarizing the participant with the procedure and the equipment.

Data recording

The data analyzed in this paper come from a larger project that also included tracking the participants’ eye movements and brain activity (which are reported elsewhere). A laptop with custom-built Prompter software was used to record the two synchronized audio channels (the original speech and the re- speaker's output). To display the respoken subtitles on the screen, we used FAB Subtitler Live. Each participant worked on his or her own voice profile in the SR software for the Polish language manufactured by Newton Technologies. The order of clips was randomized.

The test started with the equipment testing and calibration. Then, participants went through a short mock respeaking task to familiarize themselves with the procedure; the data for this task were not recorded. The respeaking test proper consisted of respeaking one clip in the intralingual condition and one clip in the interlingual condition. After each task, the participants had to answer a few questions related to his or her self-reported cognitive load.

Calculation of EVS and pauses

To calculate EVS and pauses, we used automatic time alignment (described below) to generate precise segmentation of audio with respect to the spoken words. Given the information where each word begins and ends in the audio, we computed the time difference between the utterances heard and respoken (for EVS) and the durations of gaps between words (for pauses). However, the automatic method could not be used to measure EVS in the interlingual task as it involved different languages; instead, manual alignment was used for selected words.

Time alignment procedure

In order to accurately appraise the statistics related with the timing of spoken events, an annotation known as time segmentation is required. This can be achieved either manually, which is an arduous and labor-intensive process, or using automated tools that perform a task known as automatic time alignment. Automatic time alignment can be quite precise given the right input conditions. While the quality of time alignment is difficult to assess for any given input, it relies on an automatic SR engine and thus suffers from a lot of the same problems. The minimum resolution for such systems is usually 10 ms, and provided that the right word sequence is matched correctly, the difference between the automatic and reference alignment boundaries is generally below 50 ms (Chen, Liu, Harper, Maia, & Mcroy, Reference Chen, Liu, Harper, Maia and Mcroy2004). For various pragmatic reasons, the boundary shift is not often measured in such systems and other metrics are used instead (Räsänen, Laine, & Altosaar, Reference Räsänen, Laine and Altosaar2009).

Time alignment is a well-known and thoroughly studied problem in the field of automatic speech processing (Benesty, Sondhi, & Huang, Reference Benesty, Sondhi and Huang2007; Jelinek, Reference Jelinek1997). It is usually solved using a variant of the Viterbi algorithm (Rabiner, Reference Rabiner1989), but this approach does not always work with fairly long and noisy audio sequences, as the ones studied in this paper. Instead, a technique inspired by SailAlign (Katsamanis, Black, Georgiou, Goldstein, & Narayanan, Reference Katsamanis, Black, Georgiou, Goldstein and Narayanan2011) is employed, the main idea of which is to perform simple continuous SR, which normally produces a segmentation of the words but does not guarantee their correct ordering. This initial match of words is then aligned to the original word sequence using a text-to-text alignment based on the Levenshtein-like algorithm (Levenshtein, Reference Levenshtein1966). This procedure produces a list of matches for individual words: correct, insertions, deletions, and substitutions. The correctly matched words are assumed to match with correct segments of the audio, and all the incorrect matches are realigned recursively, until convergence.

In order to produce time segmentation, the following steps were performed. The audio was recorded in a series of sessions, using a special custom-made program that can record multiple streams of audio simultaneously. Each recording comprised two audio streams: (a) the sound of the original material being respoken and (b) the audio of the respeaker's voice, recorded with a professional-grade microphone, used normally for SR purposes. The purpose of having both streams was to be able to synchronize the segmentation times later in the experiment. This was crucial to obtain accurate time delay between what the respeakers heard and what they respoke.

The time-alignment procedure requires accurate transcriptions in order to work. While this step could have been performed automatically, it was important to avoid any mistakes at such an early stage; thus this step was performed completely by hand. After human-made transcriptions in orthographic form were produced, they had to be converted to a phonetic script to perform the actual alignment. This is because the recorded speech exists only in the phonetic (i.e., spoken) form, and this is the actual information that is being aligned. For languages like Polish, grapheme-to-phoneme conversion is not very difficult, but it can produce many false positives in real-world data, which contains information like foreign words, names, or characters with no obvious phonetic representation (e.g., numbers or abbreviations; Brocki, Marasek, & Koržinek, Reference Brocki, Marasek, Koržinek, Chen, Felfernig, Liu and Raś2012).

Calculating EVS

The time-alignment described above produced a segmentation of the audio into individual word segments. This segmentation is usually generated in the form a TextGrid file used by Praat software (Boersma, Reference Boersma2002) for phonetic analysis.

To complete the analysis in the experiment, we needed to compare the sequence of segments from the reference audio (as heard by the respeaker) to the sequence of segments from the respoken audio (of individual respeakers). Since it is very likely that the respeaker will not repeat what he or she hears verbatim (Luyckx et al., Reference Luyckx, Delbeke, Van Waes, Leijten and Remael2010; Romero-Fresco, Reference Romero-Fresco2011), this comparison is not as trivial as simply matching each segment in sequence. The respeaker can omit, substitute, and in some cases even insert words that were not in the reference material (such as punctuation). Therefore, we needed to treat this problem as another candidate for the Levenshtein (Reference Levenshtein1966) algorithm. Myers's (Reference Myers1986) difference algorithm was used to find the optimal alignment of the sequences, such that the minimum number of editing operations (substitutions, deletions, and insertions) need to be performed to convert one sequence into the other. In this study, only the correctly matching segments were compared.

As the alignment for the interlingual condition could not be managed automatically, it was performed manually. Automatic alignment would require matching words between languages, which would have to rely on some form of machine translation. It was decided that such a strategy would introduce too much error into the experiment, and the manual approach was more cost effective in this case. Every 10th word from the original speech was selected for alignment with the appropriate translation equivalent from the transcribed respeaking output. If that word happened to be a function word, a cognate, or a proper name, the subsequent eligible word was selected. We thus arrived at manually aligned selected words from the interlingual condition and all automatically aligned words from the intralingual condition. In order to make the two comparable, we also selected words from the intralingual text following the same criteria we used for the interlingual condition and analyzed the data only for those words.

Calculating pauses

The pauses in respeaking follow automatically from the segmentation as presented above. Since the experiment involves a single person speaking in a quiet environment, anything that is not segmented as the words of that person can be assumed to be a pause. In other words, to compute the pauses, we simply take the difference of the beginning of one word and the end of the one preceding it. Because the participants were asked to insert punctuation marks by voicing words such as “comma” and “full stop,” it was easy to exclude from our analysis any functional pauses stemming from syntactic boundaries. We excluded all pauses following the words denoting punctuation marks. We also excluded pauses shorter than 200 ms, which is typically done to eliminate from the analysis those moments of silence which are due to the articulatory characteristics of, for example, voiceless consonants or due to the latency of the recording equipment (Warren, Reference Warren2013).

We also calculated a pause ratio in order to examine the potential influence of the pausing pattern of the original speaker on the pausing pattern of the respeaker. Pause ratio was calculated by dividing the sum of pauses in the original clip by the sum of pauses in the respoken output for each participant and each clip. If the value was 1, it meant that the total length of pauses in the original equaled the total length of pauses made by the respeaker (excluding the functional pauses after pronouncing the punctuation marks). If the ratio was smaller than 1, the respeaker made longer pauses than there were in the original clip. If the ratio was greater than 1, the total length of pauses made by the respeaker was smaller than the total length of pauses in the original clip.

EXPERIMENT 2

Method

To find out if there are any differences in EVS and pauses between different TV genres (speech, news, entertainment show, and political chat show) characterized by different speaking speeds, varying numbers of speakers and different levels of scriptedness of language, we further examined four videos respoken intralingually. We selected examples of both scripted and unscripted dialogue (Remael, Reference Remael and Diaz-Cintas2008) to find out if the character of the dialogue had an impact on the temporal characteristics of respeaking. The main difference between scripted and unscripted dialogue lies in the fact that the former is preprepared in written form to be later delivered orally, and the latter is spontaneous and does not follow any preprepared script. In linguistic terms, unscripted speech tends to be less formal and less fluent owing to more frequent pauses, hesitations, and repetitions; it also contains elements typical of spoken dialogue, certain discourse interpersonal markers (I mean, you know), vague language like hedges (kind of), references (stuff), or coordination tags (or something; Adrian, Reference Adrian2013; Quaglio, Reference Quaglio2009). From the subtitling perspective, unscripted dialogues tend to contain some irrelevant information and as such need to be “‘cleaned up’ grammatically and structurally” (Remael, Reference Remael and Diaz-Cintas2008, p. 65) in order to make an utterance more explicit. This is important in live subtitling through respeaking, because subtitles need to be easy to read and broadcasters usually “want subtitles to be written in standard language, focusing on content rather than idiosyncrasies of the speaker” (Remael, Reference Remael and Diaz-Cintas2008, p. 65).

We were also interested to see whether the number of speakers and, in consequence, overlapping speech, has any impact on EVS and pauses in respeaking. With this goal in mind, we selected videos with single speakers (speech and news) as well as those with multiple speakers, whose utterances were sometimes overlapping (entertainment show and political chat show).

We had a mixed factorial design for the intralingual respeaking task with clip type as an independent within-subject variable with four levels: speech, news, entertainment chat show, political chat show; and group as a between-groups independent variable with two levels: interpreters and bilingual controls. We expected to obtain the main effect of group; that is, we predicted that interpreters would provide respeaking with shorter EVS and shorter pauses than controls as they are used to linguistic processing constrained by similar time demands in professional interpreting assignments. We also expected the clip type to modulate EVS and pause length across participants. We predicted that the political chat show would generate the longest EVS, the longest pauses, and the smallest pause ratio due to its high speech rate and the presence of multiple speakers and overlapping speech. This would be followed by news (due to information density and the prescripted nature of the content) and the entertainment chat show (due to its medium-paced speech and multiple speakers). Speech was expected to generate the shortest EVS, the shortest pause length, and the highest pause ratio due to its slow speech rate and because it was delivered by one speaker only.

Materials

Participants were asked to intralingually respeak four clips. The clips represented different genres from Polish TV channels: prescripted speech: the New Year's address by Poland's Prime Minister Ewa Kopacz; prescripted news: an excerpt from Fakty, an evening news broadcast, reporting on miners’ protests; an unscripted entertainment chat show (Fakty po Faktach with an actress and a movie critic discussing the movie Ida after it was awarded an Oscar for best foreign language film); and an unscripted political chat show with speakers from opposite ends of the political spectrum (an excerpt from Kropka nad i, with numerous cases of overlapping speech). In this analysis, we used data from Clip 1 from Experiment 1 (speech). It turned out that the automatic analysis of EVS and pauses was impossible for Clip 4 due to numerous occurrences of overlapping speech. Therefore, further characteristics of the clips (see Table 2) and results are given for the first three clips only.

Table 2.

Characteristics of clips used in Experiment 2

Participants

The participants included two groups involved in Experiment 1, interpreters (N = 22) and bilingual controls without any interpreting experience (N = 12).

Procedure

The procedure was similar to the one used in Experiment 1. The experiment proper included intralingual respeaking of the experimental clips in a randomized order. The test ended with a short semistructured interview.

Calculation of EVS and pauses

The automatic calculation of EVS and pauses was performed similarly to that in Experiment 1. We used the data calculated for every word in the three video clips. Pause ratio was calculated as in Experiment 1.

Results

The study followed a 2 (group: interpreters, bilingual controls) × 3 (clip type) mixed factorial design. The dependent variables were: EVS, length of pauses, and the ratio of original pauses to pauses in respeaking. Table 3 presents mean results for each dependent variable and for each condition.

Table 3.

Mean (standard deviations) results for each dependent variable by condition

EVS

After visual inspection of Q-Q plots and histograms of EVS, values longer than 5000 ms were considered outliers and removed from the analysis (4.14% of data). The remaining EVS data was analyzed with LME models via the lme4 package (Bates, Reference Bates2013) within R (Baayen, Reference Baayen2008; R Development Core Team, Reference Development Core Team2010). This type of analysis combines the traditional F1 and F2 analyses or variance by treating participants and items as random effects and does not necessitate the aggregation of data over items or participants (analyses them at the trial level). The following model was fitted to the data (with sliding contrasts and random intercepts and slopes): EVS ~ group * clip_type + (1 + clip_type | subject) + (1 | item). Since sliding contrasts allowed us to look only at comparisons of adjacent levels of factors, we refitted the same model but with a different order of the levels of the variable clip type. Below we report observed rather than predicted data. The following predictors came out as significant from these analyses: EVS in chat show (M = 1829 ms) was only marginally significantly shorter than in speech (M = 2139 ms; b = 259.06; SE = 132.58; t = 1.95; p = .07). In turn, EVS in news (M = 2526 ms) was significantly longer than in speech (b = 716.84; SE = 110.44; t = 6.49; p < .001) and in chat show (b = 975.90; SE = 47.430; t = 20.63; p < .001). This suggests more cognitive effort in respeaking fast scripted speech. Even though EVS values were not significantly different across the two groups (b = −35.03; SE = 247.15; t = −0.14; p > .05), we found a significant interaction showing that the groups differed significantly in their EVS values between speech and news (b = –717.84; SE = 216.61; t = −3.31; p < .01) and between news and chat show (b = 686.73; SE = 85.85; t = 7.99; p < .001). Controls worked with a shorter EVS than interpreters both when respeaking speech and chat show. The trend reversed in the case of news, where interpreters outperformed controls by respeaking with a shorter EVS.

Pauses

Similarly to EVS data, after visual inspection of Q-Q plots and histograms of pauses, those longer than 2000 ms were considered outliers and removed from the analysis (2.9% of data). For the purposes of this analysis, all pauses were then log-transformed to normalize the distribution of the data. Below we report observed and not predicted data. We also report real means instead of log-transformed means for clarity of presentations. However, the coefficients are presented as log-transformed and calculated in the respective model. The following model was initially fitted to the data (with sliding contrasts and random intercepts and slopes): logPauses ~ group * clip_type + (1 + clip_type | subject) + (1 | item); however, it failed to converge. A model with the same set of fixed predictors and only with random intercepts did converge and yielded the following results (since sliding contrasts allowed us to look only at comparisons of adjacent levels of factors, we refitted the same model but with a different order of the levels of the variable clip type). Pauses in chat show (M = 589 ms) were significantly longer than in speech (M = 495 ms; b = 0.13; SE = 0.03; t = 4.78; p < .001) and in news (M = 502 ms; b = 0.17; SE = 0.04; t = 4.62; p < .001). The difference in pauses between speech and news was only numeric (b = 0.04; SE = 0.38; t = 1.14; p = .25). This suggests that respeaking unscripted speech produced by multiple speakers (chat show) is more challenging than respeaking single speakers, even when producing fast scripted text (news). Even though pauses were not significantly different across the two groups (interpreters: M = 534 ms; controls: M = 552 ms; b = –0.02; SE = –0.05; t = –0.43; p > .05), we found a marginally significant interaction showing that the groups differed in how much they paused when respeaking speech and news (b = 0.01; SE = 0.07; t = 1.79; p = .07). A similar interaction was found in pauses made by interpreters and controls in chat show and news (b = –0.01; SE = 0.07; t = –1.66; p = .09). This again suggests a different behavior of both groups depending on the clip. Controls paused more than interpreters when respeaking chat show and speech, while interpreters paused more when respeaking news.

Similarly to Experiment 1, we checked for correlations between EVS and pausing data in Experiment 2, and found no reliable results (r = –.003, p = .882). Furthermore, when pauses were added as a covariate to a post hoc LME model of EVS, they again failed to explain more variance (p > .05).

As in the analyses of pausing in Experiment 1, we decided to look at pause ratio to test the relationship between pauses in the respoken output in the three types of clips themselves. To that end, we calculated a pause ratio for every participant for every clip. These individual ratios were next analyzed with the following LME model: pauseRatio ~ group * clip_type + (1 | subject) with random intercept and sliding contrasts. These analyses revealed a significant effect of group where the pause ratio was significantly smaller for controls (M = 1.50) than for interpreters (M = 1.99; b = 0.54; SE = 0.26; t = 2.09; p < .05), suggesting that both groups paused less than the original speakers and that the total length of pauses relative to pauses in the to-be-respoken original was smaller for interpreters as compared to controls. This means that both interpreters and controls used pauses in the original to continue with their production of respeaking and interpreters did that to a greater extent. In addition, the pause ratio was significantly shorter for chat show (M = 1.65) than for speech (M = 2.19; b = 0.44; SE = 0.13; t = 3.25; p < .01) and marginally shorter for speech than for news (M = 1.75; b = –0.36; SE = 0.19; t = –1.98; p = .06), suggesting that the respeakers did produce respoken subtitles while using pauses, especially in the slow-paced clip (speech).