Introduction
SubtitlesFootnote 1 have been shown to support literacy (Black, Reference Black2021; Gernsbacher, Reference Gernsbacher2015; Kothari, Reference Kothari2008; Linebarger, Reference Linebarger2001), improve access to audiovisual media for deaf and hard-of-hearing viewers (Jelinek Lewis & Jackson, Reference Jelinek Lewis and Jackson2001), facilitate comprehension and foster language learning (Bisson et al., Reference Bisson, Van Heuven, Conklin and Tunney2014; Gass et al., Reference Gass, Winke, Isbell and Ahn2019; Sydorenko, Reference Sydorenko2010; Teng, Reference Teng2022; Wang & Pellicer-Sánchez, Reference Wang and Pellicer-Sánchez2022, Reference Wang and Pellicer-Sánchez2023; Winke et al., Reference Winke, Gass and Sydorenko2013; for reviews, see Kurokawa et al., Reference Kurokawa, Hein and Uchihara2024; Montero-Perez, Reference Montero-Perez2022). Despite this body of research, relatively little is still known about the temporal dynamics of subtitle reading in multimodal contexts, where viewers must integrate parallel and overlapping sources of information, including speech and soundtrack, unfolding on-screen action, and transient subtitle text.
Previous research has shown that multimodal reading-while-listening can support comprehension (Pellicer-Sánchez et al., Reference Pellicer-Sánchez, Tragant, Conklin, Rodgers, Serrano and Llanes2020, Reference Pellicer-Sánchez, Conklin, Rodgers and Parente2021) and provides some insight into attention allocation to static text presented alongside sound and static illustrations, finding no difference in visual attention to text under these conditions between native (L1) and non-native (L2) viewers. However, these studies did not investigate dynamic subtitle text and did not include a video component. Other research found that both L1 and L2 readers often read ahead of the auditory stream when static text and sound are presented concurrently (Conklin et al., Reference Conklin, Alotaibi, Pellicer-Sánchez and Vilkaitė-Lozdienė2020), a finding confirmed in subtitled video (Wisniewska & Mora, Reference Wisniewska, Mora and Levis2018). It has also been found that viewers rely more heavily on subtitles in the absence of sound or in the presence of L2 audio (de Bruycker & d’Ydewalle, Reference De Bruycker, d’Ydewalle, Hyönä, Radach and Deubel2003; Liao et al., Reference Liao, Yu, Reichle and Kruger2021, Reference Liao, Yu, Kruger and Reichle2022; Muñoz, Reference Muñoz2017; Szarkowska et al., Reference Szarkowska, Ragni, Szkriba, Black, Orrego-Carmona and Kruger2024b). In the context of language learning from subtitled videos, research showed, for example, that subtitle reading is affected by working memory capacity and native language (Gass et al., Reference Gass, Winke, Isbell and Ahn2019), as well as by age and L2 proficiency (Muñoz, Reference Muñoz2017), and that textual enhancement such as underlying, bolding, or italicizing results in increased visual attention (e.g., more, longer fixations, less skipping) and L2 learning (Lee & Révész, Reference Lee and Révész2018; Puimège et al., Reference Puimège, Montero Perez and Peters2023).
Despite this growing interest in subtitle processing, however, we still know very little about the initial stages of subtitle reading. Immediately following subtitle onset, viewers must move their eyes to a location within the subtitle area that varies with subtitle length (since most subtitles are centered, resulting in shorter subtitles being further from the left margin of the screen) and begin reading text that is externally paced and disappears dynamically, while simultaneously processing speech and visual action. More specifically, the gap concerns whether viewers orient to subtitles quickly enough to process them, whether gaze landing in the subtitle area immediately leads to reading or first requires corrective repositioning, and how these early processes are shaped by subtitle length and viewer language. This matters for streaming platforms and social media because contemporary subtitles are often short, fast, and automatically segmented. In such contexts, even a delay of a few hundred milliseconds before reading begins may determine whether a subtitle is read, partially processed, or missed altogether, especially for L2 viewers or less skilled readers.
When watching subtitled video, viewers must shift their gaze between the image and the subtitles while simultaneously processing speech. These gaze shifts are triggered by multiple factors, including the appearance of a subtitle (typically synchronized with the start of an utterance), visually salient features of the video such as shot and scene changes, and other gaze attractors such as faces, motion, color, and contrast (Smith, Reference Smith and Shimamura2013). Whatever triggers the gaze shift, it takes time: the eyes must disengage from the currently fixated screen component, move to the subtitle area, and locate an appropriate point from which reading can begin. We refer to the time required to program a saccade, move the eyes from the video area to the subtitles, and commence reading as processing latency. Processing latency becomes particularly critical in light of current subtitling practices, which increasingly allow very short minimum subtitle durations. For example, Netflix specifies a minimum subtitle duration of 5/6 of a second. Such short subtitles may disappear before viewers have time to (fully) process them, and for this reason they have been referred to as “ghost subtitles” (Kruger & Liao, Reference Kruger and Liao2025). Their existence raises questions about accessibility and readability even for skilled readers and may therefore be especially problematic for L2 readers, the population we focus on in this article.
Subtitle length is related to ghost subtitles that viewers cannot fully process before they disappear, since this kind of shorter, faster subtitle text could easily be missed altogether. We are therefore also interested in the relationship between subtitle length and early stages of subtitle processing.
Unlike static text, subtitles are transient and externally paced, and they compete with dynamic visual input for attention. Although many principles of eye-movement control established for print reading (cf. Rayner, Reference Rayner1998, Reference Rayner2009a, Reference Rayner2009b; Rayner et al., Reference Rayner, Juhasz, Pollatsek, Snowling and Hulme2005; Reichle et al., Reference Reichle, Pollatsek, Fisher and Rayner1998) apply to subtitle reading—such as when and where to move the eyes, the programming of saccades, word skipping, parafoveal processing, and effects of word frequency and word length (see Kruger & Liao, Reference Kruger and Liao2022; Krejtz et al., Reference Krejtz, Szarkowska and Łogińska2016; Liao et al., Reference Liao, Yu, Reichle and Kruger2021)—these additional constraints mean that subtitle reading is unlikely to be fully linear. Instead, it may involve corrective eye movements as viewers coordinate text processing with ongoing scene perception (Henderson & Hayes, Reference Henderson and Hayes2018; Hutson et al., Reference Hutson, Chandran, Magliano, Smith and Loschky2022), especially during the early stages of subtitle processing. This aspect of subtitle processing has not been investigated to date. If subtitle reading is not fully linear, then the time available for reading cannot be equated simply with subtitle duration. Viewers may spend part of the display time locating the subtitle and correcting their gaze position before progressive reading begins. This is important for content creators because subtitles that meet formal speed or duration guidelines may nevertheless be difficult to process if they require substantial early oculomotor adjustment.
The present study examines the initial stages of subtitle processing from the moment a subtitle appears on screen, with a focus on processing latency. Using eye tracking, we investigate how a specific subtitling feature, subtitle length, as well as viewer language (L1 vs L2), affects early subtitle reading behavior, including skipping. In doing so, the study contributes to a clearer understanding of reading in audiovisual contexts and provides evidence relevant to both psycholinguistic models of reading and current subtitling practice.
Native language and subtitle length
Investigating the difference in reading between native and non-native viewers is relevant in a highly globalized world where audiences are increasingly mixed, especially since video-on-demand services have made it much easier to watch any film with an array of linguistic options to choose from (e.g., dubbing, subtitling, voice-over, in nearly any language). This means the same materials will be consumed by different audiences who may have very different linguistic knowledge, backgrounds, and experiences.
Recent studies also indicate that intralingual subtitles (e.g., English subtitles for English videos) are increasingly favored not only by L2 viewers but also by L1 viewers, especially younger generations like Generation Z and millennials, who opt to turn them on to navigate challenges such as unfamiliar accents, poor audio quality, mumbling, or fast speech rates (Greenwood, Reference Greenwood2023; Vega, Reference Vega2023; Zajechowski, Reference Zajechowski2023). Moreover, it has been established that viewers engage with subtitles whether they need them or not, a behavior called “automatic subtitle reading” (see Bisson et al., Reference Bisson, Van Heuven, Conklin and Tunney2014; d’Ydewalle & De Bruycker, Reference d’Ydewalle and De Bruycker2007; d’Ydewalle et al., Reference d’Ydewalle, Praet, Verfaillie and Rensbergen1991), such that L1 readers can be expected to read the subtitles as well. Therefore, it is meaningful to examine L1 viewers reading L1 subtitles not only as a control group to compare L2 reading patterns to but also as a cohort in their own right, given the recent viewing trends emerging in younger L1 populations.
Both speech processing and text reading are generally considered less efficient in the L2 than in the L1 (Segalowitz, Reference Segalowitz2010), and L2 subtitled video presents L2 learners with both, thus making it an interesting medium to investigate language effects. In reading, documented differences in eye movements show L2 readers typically having more, longer fixations and shorter saccades than L1 readers, reflecting more effortful processing (see de León Rodríguez et al., Reference de León Rodríguez, Buetler, Eggenberger, Laganaro, Nyffeler, Annoni and Müri2016; Quiñonez-Beltran, Reference Quiñonez-Beltran, Seymour, Robbins, Xu and Joshi2024; Rayner Reference Rayner2009b). Furthermore, readers tend to skip fewer words and have longer reading times when reading in their L2 than in their L1 (Cop et al., Reference Cop, Drieghe and Duyck2015; Whitford & Titone, Reference Whitford and Titone2017). These patterns have found preliminary confirmation in recent research on reading subtitles (Liao et al., Reference Liao, Yu, Kruger and Reichle2022; Szarkowska et al., Reference Szarkowska, Ragni, Szkriba, Orrego-Carmona, Black, Kruger and Krejtz2024a, Reference Szarkowska, Ragni, Szkriba, Orrego-Carmona and Black2025). L2 viewers would therefore require more time than L1 viewers to read the same amount of text in a subtitle. This might be expected to produce shorter times to first fixation, as L2 viewers may attempt to reach the subtitle more quickly to ensure they can read it before it disappears. However, this effect could be cancelled out by less accurate initial processing, which might instead extend processing latency (although this has not been established empirically to date and would only be the case if these initial eye movements were under direct cognitive control). The accuracy of saccadic programming might also differ between L1 and L2 viewers, in line with the Reduced Capacity for Parafoveal Processing (ReCaPP) hypothesis presented by Fernandez and Allen (Reference Fernandez and Allen2025). According to this account, L2 readers extract less information during parafoveal processing, which limits predictive processing and may result in less efficient eye-movement planning. Although ReCaPP is formulated primarily with respect to parafoveal processing during reading, its core assumption of reduced parafoveal information uptake in L2 readers may also have implications for early oculomotor targeting at subtitle onset during the first fixation in the subtitle, potentially affecting how accurately viewers initially position their gaze during and after the first fixation in the subtitle before sustained reading begins.
The relation between subtitle language and audio language undoubtedly impacts subtitle processing. Bisson et al. (Reference Bisson, Van Heuven, Conklin and Tunney2014) compared eye movements across different soundtrack–subtitle language combinations in L1-English viewers with no knowledge of Dutch. Although their design differs from the present same-language subtitle context, their findings suggest that subtitle use varies depending on whether or how much viewers need the written text to access the spoken content. Similarly, Liao et al. (Reference Liao, Kruger and Doherty2020, Reference Liao, Yu, Kruger and Reichle2022) and Wang and Pellicer-Sánchez (Reference Wang and Pellicer-Sánchez2023) show that attention to subtitles is shaped by the relation between soundtrack language, subtitle language, and viewers’ linguistic knowledge. These studies are relevant here because they suggest that L2 viewers may rely more strongly on subtitles than L1 viewers. The present study extends this line of work by examining same-language English subtitles and by asking whether native-language status affects not only overall attention to subtitles but also the earliest stages of subtitle engagement. Despite its potential effects on text processing in audiovisual contexts (especially when it comes to ghost subtitles, as discussed above), almost no research has isolated subtitle length in their investigation of subtitle processing. Most studies to date have focused on subtitle speed, a related but distinct variable, which considers subtitle length and duration together (as it is usually calculated in number of characters per second, CPS). To our knowledge, only two studies addressed subtitle length explicitly. An earlier study by Zahedi and Khoshsaligheh (Reference Zahedi and Khoshsaligheh2021) found that longer subtitles resulted in longer processing times. Szarkowska et al. (Reference Szarkowska, Ragni, Szkriba, Orrego-Carmona and Black2025) included subtitle length (operationalized as the number of words per subtitle) in a broader multivariate analysis and showed a statistically significant effect of length on Proportional Reading Time (the percentage of time spent reading a subtitle, relative to that subtitle duration), with longer subtitles associated with higher values. No studies to date have looked at the effect of subtitle length on the initial stages of subtitle processing, such as latency.
Since longer subtitles are more visually salient, they may allow viewers to gauge the amount of reading required based on the volume of text visible in their peripheral vision before shifting their gaze to the subtitle. This increased saliency of longer subtitles may lead to shorter processing latencies, either due to bottom-up visual processing (longer subtitles are spotted more easily in the peripheral vision, which could trigger earlier gaze shifts) or because viewers recognize the larger amount of text and attempt to start reading it more quickly. In visual scene processing, for example, it has been established that saliency impacts latency, with more salient areas being fixated sooner (Anderson et al., Reference Anderson, Ort, Kruijne, Meeter and Donk2015). However, should latency be determined by top-down factors, it is possible that viewers may anticipate that longer subtitles remain on screen for a longer duration and consequently delay their eye movements towards the subtitle area to complete processing important elements of the scene before shifting their gaze down. In the present study, we explicitly put these hypotheses to the test by including an exploratory analysis of subtitle length alongside confirmatory analyses of more established L1 vs L2 language effects in a novel investigation of processing latency.
Acuity, eye movements, and attention
Although studies have shown that the locations of a small number of previously identified objects can be tracked simultaneously during a pre-attentive stage of visual processing (Pylyshyn, Reference Pylyshyn2004; Pylyshyn & Storm, Reference Pylyshyn and Storm1988), any detailed processing—such as reading or object identification—requires individual words or objects to be fixated in a strictly serial manner to allow them to be identified. This means that reading a subtitle cannot begin before the gaze has shifted to an appropriate starting point within the subtitle. Additionally, because of the distance between key visual elements on the screen and the subtitle text, the likelihood of change blindnessFootnote 2 increases. As a result, subtitles may be missed (skipped unintentionally while attention is directed somewhere else on screen, particularly in the case of very short or ghost subtitles) or deliberately skipped if another area on the screen is prioritized at that moment (intentional skipping). Conversely, this also implies that important visual changes on screen may go unnoticed while the viewer is focused on reading the subtitles.
During reading, not all words have to be fixated, as some—specifically shorter, more common or frequent words—can be identified during the fixation on the preceding word and can therefore be skipped. Longer words are more likely to be fixated and are often refixated (e.g., Rayner, Reference Rayner2009b; Rayner & McConkie, Reference Rayner and McConkie1976; Rayner et al., Reference Rayner, Sereno and Raney1996). In subtitled video, it stands to reason that longer subtitles would be less likely to be skipped purely because of their visual saliency (see Krejtz et al., Reference Krejtz, Szarkowska and Łogińska2016). Unlike in reading, however, the preview benefit (where information on a word is obtained prior to that word being fixated; see Rayner, Reference Rayner2009b, Reichle & Reingold, Reference Reichle and Reingold2013, Schotter et al., Reference Schotter, Angele and Rayner2012) does not apply to the fixation preceding the first fixation on a subtitle (i.e., the last fixation on the video before the eyes enter the subtitle area). As a result, the preview benefit cannot have an effect on the skipping of words in subtitles when the eyes first enter the subtitle. However, subtitle skipping might still occur if the subtitle goes unnoticed, disappears before the reader can start reading it, or if the viewer has already acquired sufficient contextual information and does not need to process it.
In reading languages like English, the perceptual or reading span—the range within which useful information can be obtained during a fixation—has been shown to extend roughly 3 to 4 character spaces to the left of a fixation and 14 to 15 characters to the right of the fixation. However, reading span does not extend up or down, and the preview benefit is limited to the word immediately to the right of the fixated word (see Rayner, Reference Rayner2009b for a discussion).
Eye movements and gaze control during scene perception
Subtitle reading differs fundamentally from reading static text, as it takes place within the context of dynamic scene perception (cf. Nuthmann & Henderson, Reference Nuthmann and Henderson2012, and Henderson et al., Reference Henderson, Williams, Castelhano and Falk2003, for an overview of eye movements during scene perception). Viewers must divide their attention between moving images, audio, and written language that is both temporally and spatially constrained. The addition of visual and auditory stimuli imposes further processing demands, making subtitle reading a distinctly multimodal task. As Kruger and Liao (Reference Kruger and Liao2022) note, this can lead to split attention, potentially limiting the time available for detailed processing of either the visual scene or the subtitle.
Latency
One focus of the present study is latency, where we distinguish between the time it takes a viewer to move their eyes to the subtitle (time to first fixation, TFF) and the time it takes to start reading a subtitle (or a specific part of the subtitle) after it appears, which we call processing latency. Since eye movements are motor responses, they take time to plan and execute (Rayner, Reference Rayner2009b, p. 1458). When (or if) a subtitle is detected, additional time is needed to encode its location, determine the optimal reading position, and initiate an eye movement. In reading static text, the average time required to plan and initiate a saccade is typically 175–200 ms (see Rayner, Reference Rayner2009b). Once initiated, it takes additional time to move the eyes to the intended target. This is known as saccade duration and is a function of the distance the eyes have to move. In reading, where the average saccade amplitude is around 2 degrees, saccade duration is around 30 ms, while in scene perception, where the average saccade amplitude is around 5 degrees, it ranges from 40 to 50 ms (Rayner, Reference Rayner2009b).
However, this does not account for the fact that the saccade entering the subtitle is not necessarily a reading saccade, as its primary function is to shift gaze from one activity (scene perception) to another (reading). It also does not consider the complexity of the scene being processed when the subtitle appears, which may need to be completed before shifting attention, or the varying distance between the subtitle and the preceding fixation on the image. Greater variation in saccade distance increases the likelihood of saccadic error—where the saccade misses the intended target, necessitating a corrective saccade before reading can commence (see Bowers & Poletti, Reference Bowers and Poletti2017; McConkie et al., Reference McConkie, Kerr, Reddix and Zola1988; Nuthmann et al., Reference Nuthmann, Engbert and Kliegl2005; Parker et al., Reference Parker, Slattery and Kirkby2019). For example, Parker et al. (Reference Parker, Slattery and Kirkby2019) investigated return sweeps—long leftward saccades that move the eyes from the end of a line to the beginning of the subsequent line—in the reading of multi-line static text by adults and children. They found that more than 50% of return sweeps made by adults result in “undersweepsFootnote 3 ” that require a corrective regression towards the left margin. Since the launch site of saccades from the image to the subtitle is much more variable than in the reading of multi-line text, and since subtitles vary in length and consequently in the distance of the first word from the screen margin, saccadic error in initial subtitle reading is likely to be both more frequent and less predictable, hence the focus in this study on initial corrective saccades.
Although the preferred viewing location during reading (the location where readers most frequently land on first fixation) has been shown to be located approximately halfway between the beginning and middle of a word (Rayner, Reference Rayner1979), this varies as a function of the saccade’s launch site. Specifically, when the launch site is further from the target, the landing position tends to shift leftward (Rayner, Reference Rayner2009b), except in the case of return sweeps (Parker et al., Reference Parker, Slattery and Kirkby2019). The preferred viewing location is different from the optimal viewing position, which is the center of a word and facilitates optimal word recognition (Rayner, Reference Rayner2009b). The further the saccade lands from the optimal viewing position, the higher the likelihood of a refixation (see Özkan & Acarturk, Reference Özkan and Acarturk2024). In the case of subtitle reading, due to the greater distance between the launch site on the image and the subtitle, as well as the shift from scene perception to reading, it is possible that the saccade into the subtitle area does not precisely target a specific word but rather a region in the subtitle, such as the beginning. Also, since subtitles are usually presented at the center-bottom of the screen but vary in length, the position of the text varies around the horizontal axis, resulting in changes in saccade target that are not always predictable. The gaze may, for example, land near but not at the beginning of the subtitle, increasing the likelihood that the first fixation does not mark the actual onset of reading, a possibility that we examine in our exploratory analyses. Initial fixations may therefore frequently be overshoots or undershoots (or even undersweeps—see Parker et al., Reference Parker, Slattery and Kirkby2019) or may land on the second line of a two-line subtitle, requiring small vertical corrective eye movements. These adjustments—whether in the form of refixations, regressions, or vertical corrective saccades—would contribute further to processing latency, and for these reasons we also engage in an exploratory investigation and classification of corrective eye movements in the early stages of subtitle processing.
Latency in subtitle processing
To date, very limited work has investigated the factors that influence gaze shifts between subtitle and image in subtitled video. The few studies that do address latency have typically operationalized it as time to first fixation—the interval between subtitle onset and the viewer’s first fixation within the subtitle area.
One of the earliest studies, conducted by d’Ydewalle et al. (Reference d’Ydewalle, Praet, Verfaillie and Rensbergen1991), found that hearing participants had a shorter latency to the subtitle when the sound was off, presumably because of the reliance on the subtitles in the absence of sound. Similarly, Szarkowska et al. (Reference Szarkowska, Krejtz, Kłyszejko, Wieczorek and Romero-Fresco2015) found that deaf participants had shorter TFF to subtitles than hearing viewers (209 vs 233 ms), again suggesting stronger reliance on subtitles. In the latter study, TFF was highest for verbatim subtitles (234 ms), which were longer and contained more words compared to standard (222 ms) and edited subtitles (229 ms), which contained less text. They also reported shorter TFF for subtitles presented in the standard bottom-of-screen position than for subtitles placed at the top or in mixed positions, plausibly reflecting viewers’ expectations about where subtitles typically appear.
A paper by Kruger and Liao (Reference Kruger and Liao2025) looked at processing latency in two experiments: one investigating the impact of subtitle speed (12, 20, and 28 CPS) on reading English subtitles by English L1 viewers, and another examining the influence of participants’ language (L1-Chinese, L2-English) relative to the language of the video and the subtitles on subtitle reading (no audio, English audio, and Chinese audio conditions, all with English subtitles). Unlike in previous studies, the authors redefined latency to account for the fact that the first fixation in a subtitle is not necessarily the point from which reading begins. They defined latency as the time between the subtitle onset and the first fixation followed by a forward saccade after all corrective saccades have been made to the intended target. Subtitle speed was found to have a significant impact on latency, with participants in the two conditions with faster speeds (20 and 28 CPS) displaying shorter latency than in the 12 CPS condition. Viewers shifted their gaze to the point preceding the first forward saccade significantly more quickly in the faster subtitle conditions, with latencies of 397 ms at 20 CPS and 378 ms at 28 CPS, compared to 716 ms in the 12 CPS condition. Audio language was also found to have a significant impact on latency, with Chinese audio resulting in longer latency compared to English audio or no audio. No audio and English audio resulted in latencies of 619 and 579 ms, respectively, while the latency in the presence of Chinese audio (where reliance on subtitles would be lowest) was significantly higher at 764 ms, similar to the 716 ms latency at a speed of 12 CPS in the first experiment without sound.
The current study builds on these earlier findings by taking an in-depth look at the role of viewer language group (L1-English vs L1-Polish viewers watching English subtitles in their L2) as well as the distinct contribution subtitle length may have on subtitle skipping and on latency. We distinguish between the time it takes viewers to move their eyes to the subtitles (TFF) and the time it takes them to complete initial corrective saccades and start reading (processing latency). We also focus on the causes of processing latency by investigating the types of initial corrective saccades made by viewers once they arrive in the subtitles.
Research questions and hypotheses
The present study examines early subtitle processing by disentangling three related but distinct aspects of subtitle engagement: the time it takes viewers to direct their gaze to the subtitle area (TFF), the eye-movement behavior that occurs once the gaze has landed in the subtitle area but before reading begins (processing latency), and the extent to which subtitles are skipped altogether as a function of subtitle length (skipping). The following research questions are addressed:
(1) Subtitle skipping
How do viewer language (L1 vs L2) and subtitle length (no. of characters) affect subtitle skipping?
Our hypothesis is that shorter subtitles will be skipped more often than longer subtitles (see Krejtz et al., Reference Krejtz, Szarkowska and Łogińska2016) and that overall, L1 viewers, who rely less on the written subtitle text, would skip more subtitles than L2 viewers (in part based on evidence from Liao et al. (Reference Liao, Yu, Kruger and Reichle2022) that L2 viewers pay more attention to L2 subtitles). This hypothesis is also based on the findings of Anderson et al. (Reference Anderson, Ort, Kruijne, Meeter and Donk2015), who found that more salient areas (in this case, longer subtitles) are fixated sooner than less salient areas, which could result in short subtitles being skipped because viewers cannot reach them in time.
(2) Latency
2a. Time to first fixation: How do viewer language (L1 vs L2) and subtitle length (no. of characters) affect the time it takes viewers to move their gaze to the subtitle area following subtitle onset?
2b. Processing latency: How do viewer language (L1 vs L2) and subtitle length (no. of characters) affect the time it takes viewers to start reading the subtitle after making corrective saccades?
Our hypotheses are that L2 readers, who rely more on the subtitles to support their understanding of dialogue in their second language, would have shorter TFF and shorter processing latency than L1 readers, and that longer subtitles would also result in shorter processing latency due to their visual saliency (being more visible and requiring more text to be read, in line with the findings of Anderson et al., Reference Anderson, Ort, Kruijne, Meeter and Donk2015).
(3) Corrective saccades prior to the onset of reading
What type of first saccade do viewers make within the subtitle area after gaze has landed there, and how are standard forward saccades and different types of initial corrective saccades influenced by viewer language (L1 vs L2) and subtitle length?
This analysis is exploratory because little previous research has examined the first within-subtitle saccade after gaze entry. We provisionally expected that L2 viewers might be less likely to produce an immediate standard forward saccade and more likely to produce an initial corrective saccade, given evidence that L2 readers may extract less information parafoveally than L1 readers. However, because there is little prior evidence on the specific form these initial adjustments may take in subtitle reading, we did not make directional predictions for the individual corrective-saccade types or for subtitle length.
The study includes both confirmatory and exploratory components. Research questions 1 and 2 were accompanied by directional hypotheses based on prior literature (e.g., Anderson et al., Reference Anderson, Ort, Kruijne, Meeter and Donk2015; Krejtz et al., Reference Krejtz, Szarkowska and Łogińska2016; Liao et al., Reference Liao, Yu, Kruger and Reichle2022), and analyses were pre-specified to test the main effects of subtitle length and language group. Where interactions were modelled, these were included for exploratory purposes, given plausible theoretical motivation (e.g., viewer group differences in subtitle processing dynamics). Research question 3, which examines whether and which corrective saccades are more likely to occur prior to reading onset (see sections “Corrective saccades”, “Statistical modelling”, “Initial corrective saccades”, and “Discussion”), was explicitly exploratory due to the limited prior evidence in this area. We note these distinctions in the relevant Results and Discussion sections and interpret exploratory findings with appropriate caution.
Methodology
Participants
Out of the 111 participants that initially signed up for the study, 65 were L2-English and L1-Polish speakers residing in Poland (PL), and 46 were L1-English speakers residing in the UK (UK). Overall, data from 16 participants were excluded (PL: n = 7, UK: n = 9) due to poor eye-tracking data quality or because participants were not native speakers of the languages required by the study. The final pool of usable data came from 95 participants: 37 L1 participants (UK) and 58 L2 speakers (PL). Mean age was 28.64 (SD = 9.13), ranging from 18 to 52. Gender distribution was 38 females, 19 males, and 1 non-binary participant in the L2 (PL) cohort, and 23 females, 12 males, and 2 non-binary in the L1 (UK) cohort.
Participants were asked how often they watched English-language videos with English subtitles on a 7-point scale (1 = “never,” 7 = “always”). Mean scores were 5.35 (SD = 1.92) for British participants and 6.17 (SD = 1.34) for Polish participants, indicating overall strong familiarity with this viewing mode in both groups.
The LexTALE (Lemhöfer & Broersma, Reference Lemhöfer and Broersma2012) and the online Cambridge General English proficiency testFootnote 4 were used to assess English proficiency in L2 participants. The mean LexTALE score was 79.23 (SD = 10.90), ranging from 53.75 to 100 (maximum possible score). The mean score for the Cambridge test was 22.56 (SD = 2.84), ranging from 11 to 25 (maximum possible score). Most Polish participants therefore had on average an advanced proficiency, roughly equivalent to C1/C2 of the Common European Framework of Reference for Languages (CEFR). L1 participants can also be considered proficient English readers, as they were drawn from a university population.
Video materials
Two 6–7 minute clips from the American film Don’t Look Up (2021, dir. Adam McKay) were shown with audio and same-language English subtitles. The film follows two astronomers as they strive to warn the world about the imminent catastrophic impact of a comet with Earth.
There were 228 subtitles in total (99 in clip 1 and 129 in clip 2), with an average subtitle speed of 16.25 CPS, ranging from 4 to 27.65 CPS. The minimum subtitle duration was 1 second, while the maximum duration was 6 seconds. Due to a coding error, two subtitles were excluded, resulting in a final set of 226 subtitles for analysis.
Since participants were either L1-English speakers or highly proficient L2-English speakers, we did not purposefully seek “easier” materials (simple plot, slow pace, low lexical density, low-complexity visuals, etc.) but instead chose authentic video that was at times fast-paced, contained specialized terminology (science, astronomy, politics), and included multiple speakers, thus prioritizing ecological validity. The two clips were selected because they were self-contained and captured two key related moments in the video plot (namely the discovery of the deadly comet by a professor and his student and their visit to the White House to discuss solutions to avert a catastrophe). Table 1 below presents information about the two clips extracted from the video, showing that both subtitle speed and text readability were comparable.
Characteristics of the experimental video materials
Table 1. Long description
A table comparing characteristics of two video clips. The table has 2 rows and 8 columns. Column headers are Clip ID, Clip duration (min: sec), No. of subtitles, No. of words, No. of sentences, Average subtitle speed in CPS (SD), Coh-metrix L2 reading index, and Coh-metrix fleisch reading ease. Row 1: Part 1, 06:56, 99, 672, 109, 16.06 (4.91), 21.749, 83.616. Row 2: Part 2, 06:10, 129, 929, 124, 16.39 (5.20), 21.742, 82.799.
Readability indices were calculated for each clip as a whole (all subtitle text combined) using the default readability metrics in the Coh-Metrix tool (Graesser et al., Reference Graesser, McNamara, Cai, Conley, Li and Pennebaker2014; https://soletlab.asu.edu/coh-metrix/). We report the Flesch Reading Ease index as a widely used readability measure and the Coh-Metrix L2 Reading Index because it is specifically designed to estimate readability for second-language readers (Crossley & McNamara, Reference Crossley and McNamara2011). The indices were included solely to demonstrate that the two clips were comparable in readability rather than to characterize the readability of individual subtitles.
Equipment
Eye-tracking data were collected using two EyeLink eye trackers (SR Research, Canada): a Portable Duo in the UK and a 1000 Plus in Poland. In the UK, the Portable Duo tracker was mounted on an ASUS GX701L laptop. Participants sat roughly 60 cm from its display screen (1920 × 1080 resolution) wearing a target sticker for remote tracking. In Poland, the EyeLink 1000 Plus tracker was used with an ASUS VG248QE (16:9) HD monitor screen (1920 × 1080 resolution). Participants sat roughly 80 cm from this screen, and a chinrest was used to mitigate head movement. In both experimental setups, each subtitle letter subtended ∼0.32° of the visual angle, ensuring comparability. Following the manufacturer’s recommendations, a 13-point calibration was performed with the Portable Duo, and a 9-point calibration with the 1000 Plus. Eye movements were recorded binocularly at 1000 Hz in all cases, and data are analyzed for the right eye only.
Experimental procedure
Testing was performed individually in a single session in each of the institutions where data were collected. Each institution obtained separate ethical approval before the study commenced. Upon arrival in the lab, all participants signed a consent form and completed a demographics questionnaire, after which the L2 participants took the L2-English proficiency tests. While their eye movements were being tracked, viewers watched two clips taken from the same video, each immediately followed by a recall questionnaire (not analyzed here). After watching, participants also completed a 15-minute interview and a working memory task (also not analyzed here). For details about the demographics, recall questionnaire, and working memory test, see Szarkowska et al. (Reference Szarkowska, Ragni, Szkriba, Orrego-Carmona and Black2025). At the end of the experiment, participants were rewarded with a bookstore voucher.
Data analysis procedure
The present study analyses eye movements from a saccade report generated via the EyeLink data analysis software Data Viewer (SR Research, Canada), with two interest areas, one on the video area and one on the subtitle area. Interest periods were set to follow subtitle timing (each subtitle was a separate interest area corresponding to the subtitle duration). Below, we review the variables analyzed in each research question (skipping, latency, and corrective saccades) and describe the statistical models used.
Skipping
The first research question examines subtitle skipping, which provides an indication of the extent to which participants miss or decide not to read entire subtitles. A subtitle is considered skipped when no saccades were made that started or ended in that subtitle (provided that there were valid eye movements on the screen during the presence of the subtitle).
Latency
The second research question examines latency, which we operationalize in two ways. To enable comparison with previous studies (d’Ydewalle et al., Reference d’Ydewalle, Praet, Verfaillie and Rensbergen1991; Szarkowska et al., Reference Szarkowska, Krejtz, Kłyszejko, Wieczorek and Romero-Fresco2015, Kruger & Liao, Reference Kruger and Liao2025), latency is defined first as time to first fixation (TFF; see sections “Latency”, “Latency in subtitle processing”, and “Research questions and hypotheses”). To examine in more detail whether corrective movements occur before forward subtitle reading plausibly begins, we also examine processing latency. To define this variable, we identified the first saccade that started and ended in the subtitle. If that saccade was forward and had an amplitude larger than 1 degree of visual angle (roughly 3 characters), we consider the start of the first fixation followed by such a standard forward saccade as the point where processing commenced, such that processing latency would then be the time between the onset of the subtitle and the start of the fixation preceding such a standard forward saccade. This is based on the fact that, in reading research, the average amplitude of a within-word corrective saccade is around 3 to 4 characters (cf. McConkie et al., Reference McConkie, Kerr, Reddix, Zola and Jacobs1989; Vergilino-Perez et al., Reference Vergilino-Perez, Collins and Doré-Mazars2004), although this is influenced by word length. Using a boundary of 3 characters to distinguish a standard forward saccade from refixations was therefore considered a conservative choice to avoid inflation of corrective saccades.
However, some forward saccades may also be triggered by the need to make a corrective saccade to an optimal viewing position where the viewer would like to start reading. We therefore first attempted to distinguish between forward saccades that were likely to be under the cognitive control of viewers (standard forward saccades) and forward saccades that were likely to be corrective in nature (corrective forward saccades). In reading research, mislocated fixations that miss their intended target are considered to be corrected by immediate initiation of a new saccade program, which results in shorter fixations (see Nuthmann et al., Reference Nuthmann, Engbert and Kliegl2005). Nuthmann and colleagues use 125 ms as the latency of the error-correcting program, although they also consider corrective saccade latencies to be even shorter based on previous research. In other words, if a fixation is very short, it is likely because a saccade program was executed as soon as possible after arriving at the target location to correct misplacement. In simulations, researchers often set the latency for a corrective saccade to around 120–150 ms to match observed data. This captures the empirical fact that corrective refixations happen faster than ordinary, cognition-driven eye movements (see Yang & McConkie, Reference Yang and McConkie2001).
Since we did not want to impose an overly strict a priori cut-off, we derived a data-informed temporal criterion for classifying short-latency forward saccades as likely corrective. For this, we analyzed the duration of the first fixation in the subtitle area. We first recoded the first within-subtitle saccade as forward versus non-forward, where forward saccades were rightward saccades with an amplitude of at least 1 degree, and non-forward saccades included refixations, regressions, and vertical corrections. We then fitted a binomial logistic regression to determine whether first fixation duration predicted the likelihood that the first within-subtitle saccade was forward rather than non-forward. The model was statistically significant (p < .001), indicating that first fixation duration predicted the likelihood of a forward saccade rather than a non-forward saccade. The coefficient for fixation duration was positive and significant (Est. = 0.008, SE = 0.0003, p < .001), suggesting that longer fixations were associated with an increased likelihood that the first within-subtitle saccade was forward rather than non-forward.
We subsequently fitted a segmented logistic regression model to identify whether there was a change point in this relationship. This segmented regression yielded a breakpoint in fixation duration at approximately 177.41 ms (SE = 2.17). This breakpoint does not directly distinguish standard forward saccades from corrective forward saccades, because the distinction was introduced only after the breakpoint analysis. Rather, it identifies a change point in the relationship between fixation duration and the probability that the first within-subtitle saccade was forward rather than non-forward.
We then used this empirically estimated change point, together with prior oculomotor evidence that very short fixations are often associated with rapid corrective saccade programming, as an operational threshold. Forward saccades preceded by first fixations shorter than 177 ms were classified as likely corrective forward saccades, whereas forward saccades preceded by fixations at or above the breakpoint were classified as standard forward saccades. This classification should therefore be understood as a data-informed operational distinction designed to avoid treating rapid corrective movements as evidence of progressive reading onset, rather than as a direct observation of the psychological status of each forward saccade.
First, within-subtitle saccades were therefore classified as corrective if they did not meet the criteria for standard forward saccades. When the first within-subtitle saccade was classified as a standard forward saccade, processing latency was calculated as the time between subtitle onset and the start of the fixation preceding that saccade. When the first within-subtitle saccade was classified as corrective, processing latency was calculated as the time between subtitle onset and the start of the fixation following that corrective saccade. This operationalization therefore incorporates the immediate cost of correcting saccadic error after gaze enters the subtitle area while avoiding the assumption that all subsequent subtitle-internal saccades before sustained forward reading necessarily reflect further correction. Given the more dynamic and potentially non-linear nature of subtitle reading, later non-forward saccades may also reflect visual sampling, checking, or local search rather than a simple continuation of the initial corrective process.
Corrective saccades
The third research question examines initial corrective saccades, defined here as the first saccade that both started and ended within the subtitle area after gaze had entered the subtitle and that does not meet the requirements for a standard forward saccade. This operationalization was chosen to isolate the immediate oculomotor adjustment at subtitle entry. It does not capture the full sequence of possible corrective, sampling, or search-like saccades that may occur before sustained forward reading begins. Operationally, we classified the first saccade within the subtitle as progressive and reading-driven (i.e., a standard forward saccade) only if it was forward, had an amplitude greater than 1 degree of visual angle, and was preceded by a fixation longer than the 177 ms fixation-duration breakpoint defined above. All other first-within-subtitle saccades were treated as initial corrective saccades. Under this classification, a standard forward saccade suggests that viewers begin progressive reading immediately upon entering the subtitle area; otherwise, the first saccade is interpreted as corrective repositioning towards the optimal viewing position.
For the exploratory analyses, we distinguished the following types of initial corrective saccades to characterize eye-movement patterns at subtitle onset:
-
Refixations: horizontal saccades shorter than 1 degree of visual angle that correct an undersweep, undershoot, or overshoot, as well as saccades that were so short in amplitude that direction could not be classified.
-
Regressions: horizontal saccades to the left with an amplitude of equal to or more than 1 degree (to exclude refixations that are likely to be within-word).
-
Corrective vertical saccades: saccades within the subtitle area that move the eyes up or down and therefore shift the gaze between different lines.
-
Corrective forward saccades: forward saccades longer than 1 degree of visual angle, preceded by a short fixation (based on the breakpoint discussed above).
The 1-degree boundary (approximately 3 subtitle characters, with one character subtending 0.32 degrees of visual angle) was chosen to distinguish corrective forward saccades and regressions from refixations. In reading, standard forward saccades typically span between 7 and 9 character spaces (approximately 2 to 3 degrees of visual angle; Inhoff et al., Reference Inhoff, Solomon, Radach and Seymour2011; Rayner, Reference Rayner1998). Forward saccades shorter than 1 degree of visual angle are therefore unlikely to reflect progressive reading and are more likely to represent local corrective adjustments towards an optimal viewing position. Likewise, regressions shorter than 1 degree of visual angle are unlikely to represent inter-word regressions and were therefore classified as refixations.
Statistical modelling
We analyzed the data using mixed-effects models to match the measurement scale and distribution of each dependent variable while accounting for the hierarchical structure of the data (multiple first-saccade observations per participant and per subtitle). Language (L1 vs L2) was treated as a between-participant factor, and subtitle length (in characters) as a within-participant predictor.
Analyses addressing research questions 1 (subtitle skipping) and 2 (latency) were hypothesis-driven and focused on testing pre-specified main effects of subtitle length and language group. Where interaction terms between language and subtitle length were included, these were modelled for exploratory purposes, motivated by theoretical plausibility rather than a priori predictions. Research question 3, which concerns the question of whether and which initial corrective saccades are made prior to reading onset, was explicitly exploratory due to the absence of established hypotheses in the literature. Throughout the manuscript, exploratory findings are signposted as such and interpreted with appropriate caution.
For subtitle skipping (binary outcome), we used a sequence of approaches to characterize and model non-linearity: an initial segmented generalized linear model (GLM) to identify a candidate breakpoint, a binomial generalized linear mixed-effects model (GLMM) with a natural cubic spline for subtitle length to capture smooth non-linear effects with partial pooling by participant and subtitle, and a confirmatory GLMM with a manual breakpoint at the value indicated by the segmented analysis.
Continuous latency outcomes (time to first fixation; processing latency) were modelled with linear mixed-effects models (LMMs), using robust estimation where residual diagnostics suggested non-normality or heteroscedasticity.
The initial saccade type (standard forward, corrective forward, refixation, regression, or vertical corrective saccade) was analyzed using a Bayesian multilevel multinomial logistic regression implemented in the brms package. This approach accommodates the multinomial outcome while accounting for repeated observations by including random intercepts for participants and subtitles and participant-level random slopes for subtitle length. Bayesian estimation was chosen because it naturally supports multilevel multinomial models with partial pooling across participants and items and provides full posterior estimates of parameter uncertainty.
All analyses were executed in R (version 4.4.2), and full details about the modelling (including packages used, model selection, and diagnostics) are available as supplementary materials.
Results
Skipping
Question 1 asked whether viewer language and subtitle length affect subtitle skipping. Participants skipped very few subtitles overall (see Table 2). Mean skipping rates per participant were 6.00% (SD = 9.16) for L1 viewers and 4.69% (SD = 7.87) for L2 viewers, indicating that both groups looked at the vast majority of subtitles.
Subtitle skipping by language
Table 2. Long description
A table comparing subtitle skipping percentages by language with two rows and four columns. The columns are labeled Language, Mean skipping (percent), SD, and N. The row labels are L1 and L2. Row 1: L1, 6.00, 9.16, 37. Row 2: L2, 4.69, 7.87, 58.
Note: Skipping refers to entirely skipped subtitles averaged over participant and language.
To explore whether skipping varies with subtitle length, we first visualized the percentage of skipped subtitles binned at 5-character intervals (Figure 1). The bin plot suggested a non-linear relationship: skipping was relatively higher for very short subtitles and declined with length in both language groups, with L1 and L2 differences most apparent for very short subtitles (under 5 characters).
Subtitle skipping per participant by language and binned subtitle length.
Note: Bin labels use half-open intervals: the parenthesis on the left indicates the lower bound is excluded, and the square bracket on the right indicates the upper bound is included (e.g. (0, 5] denotes subtitle lengths > 0 and ≤ 5 characters).
For formal analyses, we proceeded in three steps. First, an exploratory segmented (piecewise) GLM without random effects identified a candidate breakpoint in the length–skipping relationship (cf. Muggeo, Reference Muggeo2003, Reference Muggeo2008 for a discussion of segmented linear regression). This model suggested a breakpoint at approximately 15.3 characters (uncertainty ≈ 1.57). In this exploratory GLM, both language and subtitle length were significant predictors (ps < .01). However, because the model did not account for repeated measures by participant and subtitle item, these significance levels may be inflated.
Second, to model the non-linear effect while accounting for the nested structure, we fitted a binomial GLMM with a natural cubic spline (three basis functions) for subtitle length and random intercepts for participants and subtitles (cf. Harrell, Reference Harrell2015). The spline model is exploratory and descriptive. We used a natural cubic spline with 3 degrees of freedom (df) to capture potential curvature in the relationship between the predictor (e.g., subtitle length) and the probability of skipping. A spline with 3 degrees of freedom fits a piecewise smooth curve that can bend once, allowing for one inflection point where the direction of the slope can change. This makes it more flexible than a straight line (1 df) or a simple quadratic (2 df), but still parsimonious enough to avoid overfitting, especially with limited data per bin or level. All three spline terms were highly significant (ps < .001), indicating a robust non-linear effect of subtitle length on skipping. In this spline GLMM, the main effect of language was marginal (p = .068), and there was no evidence of an interaction between language and subtitle length, indicating that both groups showed a similar non-linear response to length.
Third, to provide an interpretable piecewise effect, we fitted a confirmatory GLMM with a manual breakpoint at 15 characters based on the inflection point identified in the earlier segmented GLM (estimated via the segmented package; Muggeo, Reference Muggeo2003). This threshold reflects the point where subtitle skipping began to decline more gradually with increasing length. The model included two piecewise linear predictors representing subtitle length before and after the breakpoint, fixed effects for language group (L1 vs L2), the two language-by-length segment interactions, and random intercepts for participant and subtitle.
This approach allowed us to directly test whether the effect of subtitle length on skipping differed before and after the breakpoint and whether these effects varied by language group. Unlike segmented regression models, which estimate the breakpoint from the data (Muggeo, Reference Muggeo2003), this manual breakpoint GLMM treats the threshold as known (based on exploratory modelling) and allows for hypothesis-driven group comparisons using a standard mixed-effects framework.
Results indicated that skipping decreased significantly as subtitle length increased, both below (b = −0.15, SE = 0.03, z = −4.65, p < .0001) and above (b = −0.03, SE = 0.005, z = −6.23, p < .0001) the character breakpoint, with a steeper decline for short subtitles and a flatter slope thereafter. There was no significant group difference in subtitle skipping probability at the reference level (subtitles with a length of 0 on the centered scale), b = –0.80, SE = 0.53, z = −1.51, p = .12. Importantly, group comparisons at other subtitle lengths are captured by the interaction terms, neither of which were significant. This suggests that L1 and L2 viewers showed similar patterns of skipping across the length range.
While model comparison (AIC) suggested that the spline GLMM provided a marginally better fit than the manual breakpoint GLMM, the latter offers a more directly interpretable threshold for understanding how skipping behavior changes with subtitle length. Figure 2 illustrates the predicted probability of skipping subtitles as a function of subtitle length, separately for L1 vs L2 speakers. The dashed vertical line at ∼15 characters marks the breakpoint detected in the segmented regression analysis and confirmed across the skipping models.
Predicted probability of skipping from the GLMM with a manual breakpoint.
Note: Solid and dashed lines indicate predicted probability. Dashed vertical line indicates the breakpoint at ∼15 characters.
In answer to Question 1, subtitle skipping decreased with increasing length, with a steep decline for very short subtitles that levelled off after around 15 characters. Although L1 viewers show slightly higher skipping rates (Table 2), once participant- and item-level variability are accounted for, the effect of language is small and not statistically significant.
Time to first fixation (TFF)
Question 2a asked whether viewer language and subtitle length affect TFF. As can be seen in Table 3 below, TFF for both language groups was around 400 ms, with L2 viewers (M = 422 ms) taking slightly longer to get to the subtitle area compared to L1 viewers (M = 397 ms).
Time to first fixation by language group
Table 3. Long description
A table comparing time to first fixation (TFF) by language group. The table has three columns and two rows. Column headers are Language, Mean TFF (ms), SD (ms), and N. Row labels are L1 and L2. Row 1: L1, 397.32, 105.04, 37. Row 2: L2, 422.17, 164.90, 58.
Note: TFF = Time to first fixation; SD = Standard Deviation; N = Number of participants.
To meet model assumptions, we analyzed the square-root transformed TFF. A linear mixed-effects model predicting √TFF was fitted with fixed effects of subtitle length (characters), language group (L1 vs L2), and their interaction, and random intercepts for participants and subtitles plus a by-participant random slope for subtitle length. Results indicated a robust effect of subtitle length on √TFF (b = 0.73, SE = 0.21, t(173.45) = 3.46, p < .001), such that longer subtitles were associated with larger √TFF. The coefficient for language group (b = 0.51, SE = 0.60, t(92.49) = 0.85, p = .40) represents the estimated group difference at mean subtitle length (0 on the scaled variable) and was not significant. Crucially, the interaction between subtitle length and language group was significant (b = 0.46, SE = 0.22, t(90.06) = 2.09, p = .040), indicating that the impact of subtitle length on √TFF was greater for L2 viewers. In other words, increasing subtitle length produced a steeper rise in TFF for L2 viewers than for L1 viewers. Random effects indicated substantial variability across participants and subtitle items. The correlation between the participant intercept and participant slope for subtitle length was high (r = .88), suggesting individual differences in baseline TFF and in sensitivity to subtitle length. For interpretability, predicted √TFF values from the model were back-transformed to milliseconds when plotting (Figure 3). These back-transformed predictions show that the difference in TFF between short and long subtitles is modest in absolute ms but larger for L2 viewers (see Figure 3 for predicted values and 95% CIs on the original scale).
Predicted time to first fixation as a function of subtitle length and language group.
Note: Lines show model predictions back-transformed to milliseconds (predicted TFF = [predicted √TFF]2); shaded ribbons show 95% confidence intervals. Points show observed means (binned) for display only. The interaction indicates a steeper increase in TFF with subtitle length for L2 than for L1.
Processing latency
As explained earlier, processing latency can be defined as the time between the onset of the subtitle and the time reading is likely to have commenced. If the viewer starts reading immediately following the first fixation (characterized by a standard forward saccade), this processing latency is calculated as the time between subtitle onset and the start of the fixation preceding the first standard forward saccade. If the first saccade in the subtitle is a corrective saccade, then processing latency is calculated as the time between subtitle onset and the start of the fixation following a corrective saccade (see section “Data analysis procedure”). As can be seen in Table 4, the mean fixation duration of fixations preceding corrective saccades is consistently lower than those preceding standard forward saccades.
Descriptive summary of first fixation duration by reclassified saccade type
Table 4. Long description
A table comparing saccade types with mean first fixation duration, mean saccade amplitude, and saccade frequency. The table has five rows and four columns. The columns are labeled Saccade type, Mean FFD, Mean saccade amplitude, and Saccade frequency. The rows are labeled with different saccade types: Standard forward, Corrective forward, Refixation, Regression, and Vertical. Each row provides values for Mean FFD, Mean saccade amplitude, and Saccade frequency for the corresponding saccade type. For example, the Standard forward saccade type has a Mean FFD of 230, a Mean saccade amplitude of 3.43, and a Saccade frequency of 3952.
Note: Mean FFD is calculated in milliseconds, saccade amplitude in degrees of the visual angle, and saccade frequency in number of occurrences.
Question 2b asked whether viewer language and subtitle length affect processing latency (time from subtitle onset to likely reading onset). Descriptive statistics in Table 5 show that L2 viewers have very similar latencies (M = 566 ms) to their L1 counterparts (M = 569 ms). On average, processing latency exceeded TFF by approximately 170 ms for L1 and 150 ms for L2 (cf. Table 3).
Processing latency by language group
Table 5. Long description
A table comparing processing latency by language group. The table has three columns and two rows. The columns are labeled Language group, Mean latency (ms), SD, and N. The rows are labeled L1 and L2. Row 1: L1, 569.02, 118.55, 37. Row 2: L2, 566.26, 179.33, 58.
A linear mixed-effect model predicting log-transformed processing latency was fitted with fixed effects for language (L1 vs L2), subtitle length (in characters, scaled), and their interaction. There was no significant difference in processing latency between L1 and L2 readers at the mean subtitle length (b = −0.0046, SE = 0.0451, t(94.19) = −0.103, p = .918). However, subtitle length was significantly and positively associated with processing latency (b = 0.0424, SE = 0.0160, t(179.03) = 2.65, p = .009), indicating that longer subtitles tended to delay the initiation of reading. Because the interaction between language group and subtitle length was not significant (b = 0.0243, SE = 0.0168, t(91.96) = 1.44, p = .152), this effect reflects a general increase in latency with subtitle length that does not differ reliably between L1 and L2 groups. Random effects showed substantial variability in both intercepts and slopes across participants, with a strong positive correlation (r = .74) between individual baseline latency and subtitle length sensitivity. Figure 4 presents the modelled effects with predictions back-transformed to milliseconds for interpretability.
Predicted processing latency by subtitle length and language.
Initial corrective saccades
Question 3 is exploratory and asks whether, out of all first-within-subtitle saccades, standard forward saccades (reading-driven) or various types of corrective saccades were more probable, and what types of corrective saccades viewers make before reading begins. We examined whether language group, subtitle length, and their interaction predicted initial saccade type. Table 6 shows the breakdown of initial saccade types by language group.
Initial saccade types: mean percentages by language group
Table 6. Long description
A table comparing mean percentages of initial saccade types by language group. The table has five rows and three columns. The columns are labeled ‘Saccade type’, ‘L1 (SD)’, and ‘L2 (SD)’. The rows are labeled with different saccade types: Standard forward, Corrective forward, Refixation, Regression, and Vertical. Each row contains values for L1 and L2 with their respective standard deviations. Row 1: Standard forward, L1 23.5 (10.2), L2 28 (12.1). Row 2: Corrective forward, L1 36.4 (13.4), L2 28.1 (12.5). Row 3: Refixation, L1 4.4 (3.2), L2 7.7 (5). Row 4: Regression, L1 27.4 (11), L2 28.7 (11.9). Row 5: Vertical, L1 8.5 (5.3), L2 7.7 (3.9).
Note: Percentages reflect the mean proportion of each saccade type per participant. SD = Standard Deviation.
Bayesian multilevel model. The initial saccade type was analyzed using a Bayesian multilevel multinomial model with standard forward saccades as the reference category. The model included random intercepts for participant and subtitle and participant-level random slopes for subtitle length. Odds ratios were obtained by exponentiating the posterior means of the log-odds coefficients. L2 readers were less likely than L1 readers to produce initial corrective forward saccades (OR = 0.63, 95% CrI [0.45, 0.86]) and more likely to produce initial refixations (OR = 1.51, 95% CrI [1.01, 2.25]). There was no credible effect of language group on regressions, and a trend (not fully credible) toward lower odds of vertical corrective saccades in L2. Each unit increase in subtitle length was associated with lower odds of refixation (OR = 0.98, 95% CrI [0.97, 0.98]) and vertical corrective saccades (OR = 0.96, 95% CrI [0.95, 0.97]). LOO model comparison between the Bayesian interaction model and the Bayesian additive model indicated no evidence that the language × subtitle length interaction improved predictive performance (elpd_diff = −0.8, SE = 1.4). We therefore retained the simpler Bayesian additive model for interpretation.
Predicted probabilities from the Bayesian model are visualized in Figure 5 as a function of subtitle length for L1 and L2 readers using posterior predictions. These plots help interpret the non-linear patterns of saccade classification across subtitle lengths and language groups.
Predicted probabilities of initial saccade types as a function of subtitle length and language group.
Note: Posterior predictions; lines are median predictions, shaded ribbons = 95% CrI (credible interval).
Figure 5. Long description
The image contains five line graphs showing predicted saccade type probabilities as a function of subtitle length for two language groups, L1 and L2. Each graph represents a different saccade type: Forward, Forward_Corrections, Refixation, Regression, and Vertical. The x-axis for all graphs is labeled ‘Subtitle Length (Characters)’ and ranges from 0 to 80. The y-axis is labeled ‘Predicted Probability’ and ranges from 0 percent to 50 percent. The lines represent median predictions, with shaded ribbons indicating 95 percent credible intervals. In the Forward graph, both L1 and L2 show an increasing trend in predicted probability as subtitle length increases. In the Forward_Corrections graph, L1 shows a slight increase, while L2 shows a relatively stable trend. The Refixation graph shows a decreasing trend for both L1 and L2 as subtitle length increases. The Regression graph indicates an increasing trend for both language groups. The Vertical graph shows a decreasing trend for both L1 and L2. The graphs are used to compare the predicted probabilities of different saccade types across varying subtitle lengths for the two language groups.
Discussion
The present study examined subtitle skipping and saccadic eye movements during the initial stages of subtitle processing. Specifically, we investigated how language group (L1 vs L2) and subtitle length (measured as the number of characters) affect subtitle skipping and latency, operationalized both as the time between subtitle onset and the first fixation made in the subtitle area (TFF) and the time it takes viewers to actually start reading the subtitle (processing latency). We also classified the types of first corrective saccade that are made before reading can commence.
Research question 1 asked whether subtitle length and viewer language affect subtitle skipping. Subtitle length had a robust effect on subtitle skipping, with a clear breakpoint around 15 characters whereby subtitles shorter than this threshold were significantly more likely to be skipped than longer ones. For subtitles longer than 15 characters (less than half a line), length continued to predict skipping, but the effect attenuated. Language group did not significantly predict skipping once participant- and item-level variability were accounted for, although descriptive means indicate that L1 viewers skipped slightly more subtitles than L2 viewers (presumably because of a reduced need for subtitles). These results suggest that skipping of very short subtitles is a general, stimulus-driven behavior that is largely independent of language background (at least at the advanced proficiency level examined here) and as such likely reflects early, low-level perceptual or oculomotor constraints. Viewers could therefore be much more likely to skip or miss short subtitles due to the demands of visual processing. These findings raise concerns about the readability of very short subtitles displayed for short periods. The fact that short subtitles—particularly under 15 characters—may disappear before reading can commence at all is especially relevant in light of current platform practices. For example, Netflix sets minimum subtitle duration at 5/6 of a second, as discussed in the Introduction, which may inadvertently increase the likelihood of missed or “ghost” subtitles (Kruger & Liao, Reference Kruger and Liao2025), even for skilled readers. Although native language did not predict skipping in the current study, it cannot be excluded that, with less proficient L2 cohorts, current subtitle practices would lead to an increase in ghost subtitles, negatively impact comprehension, and potentially diminish some of the benefits of L2 subtitles discussed at the beginning of this article.
Research question 2 asked whether subtitle length and viewer language affect TFF and subsequent processing latency. Contrary to both hypotheses, L2 viewers did not show shorter latencies than L1 viewers, and all viewers did not rush to move from the images to the subtitle area when there was more text to read. TFF increased instead with subtitle length; the interaction between length and language group was significant (the increase in TFF with length was larger for L2 than L1), and processing latency likewise increased with subtitle length (no main effect of or interaction with language group). Thus, rather than showing faster processing, L2 viewers were more sensitive to increases in subtitle length at the orienting stage, and longer subtitles produced longer, not shorter, processing latencies in both groups. The interaction between subtitle length and viewer group was not part of our original hypotheses and should therefore be interpreted as exploratory rather than confirmatory.
Our analyses of latency showed related but distinct effects at two stages of initial subtitle processing. For TFF, both language groups took approximately 400 ms (397 and 422 ms in L1 and L2, respectively) to shift their gaze to the subtitle following its onset. Subtitle length was associated with increased TFF overall, and the significant interaction with language group (see Figure 3) suggests that L2 viewers may experience a steeper increase in latency for longer subtitles. This trend points to a greater sensitivity to visual or linguistic load in L2 viewers, which could be further investigated in future studies. For processing latency (the time until reading is likely to have started), the average latency was around 570 ms (569 and 566 ms in L1 and L2, respectively), and processing latency also increased with subtitle length. There was no reliable main effect of language group or interaction between language group and length. This is an important finding, as it helps to explain the skipping of shorter subtitles by participants in both language groups: by the time viewers were ready to start reading, these subtitles—particularly ones under 15 characters—may already have disappeared, or the viewer has left too little time for complete processing. The increase in processing latency with length was contrary to our prediction (which hypothesized shorter latencies for longer, more salient subtitles). One plausible interpretation is that viewers can evaluate subtitle length peripherally and adapt their oculomotor strategy accordingly, delaying gaze shifts toward longer subtitles because those subtitles will stay on screen for longer and therefore afford more time for foveal reading, which may reduce the urgency of immediate orienting. Whatever the mechanism, the net result is that longer subtitles tended to prolong both attention orienting and processing latency stages, and L2 viewers were particularly affected at the orienting (TFF) stage.
Addressing research question 3, we move to an explicitly exploratory analysis of the first saccade made within the subtitle area after gaze had entered the subtitle. This analysis was designed to characterize the immediate oculomotor adjustment at subtitle entry, rather than the full sequence of eye movements that may occur before sustained reading begins. Relative to initial standard forward saccades, L2 viewers are more likely than L1 viewers to make small horizontal refixations and less likely to make corrective forward saccades with a weaker tendency towards fewer vertical corrective saccades. Subtitle length also shapes these early adjustments: as subtitles get longer, refixations and vertical corrective saccades become less likely in both language groups. These results show that the spatial characteristics of the first within-subtitle saccade differ systematically with viewer language and with subtitle length. However, because the analysis was restricted to the first within-subtitle saccade, we do not interpret the findings as evidence about the full sequence of pre-reading corrective movements.
This exploratory analysis of initial corrective saccades, which focused on the first saccade made within the subtitle, revealed fine-grained systematic differences that were not fully captured by global measures such as skipping or latency. The pattern described above is consistent with a ReCaPP-informed interpretation hypothesis (Fernandez & Allen, Reference Fernandez and Allen2025), but it should be treated cautiously because the present analysis does not directly test parafoveal prediction or the full sequence of pre-reading corrections. ReCaPP proposes that L2 readers have reduced capacity for parafoveal processing, which may affect the efficiency of eye-movement planning during reading. In the present subtitle context, one possibility is that reduced or different use of coarse parafoveal information contributes to group differences in the first within-subtitle adjustment. However, the functional meaning of the observed pattern is not unambiguous. A higher rate of refixations in L2 viewers could reflect more cautious local adjustment or weaker parafoveal guidance, but it could also indicate that the initial landing position was already close enough to a useful viewing location to require only a small correction. Similarly, the higher rate of corrective forward saccades in L1 viewers may reflect a larger initial spatial adjustment rather than more efficient parafoveal processing. We therefore interpret the ReCaPP account as a plausible but provisional explanation: the present findings show that L1 and L2 viewers differ in the form of their earliest subtitle-internal adjustment, but they do not establish whether one group’s adjustment was more efficient than the other’s. Future work could test this interpretation more directly by analyzing the complete sequence of subtitle-internal saccades before the first standard forward saccade, including the number, amplitude, direction, and timing of corrective movements prior to reading onset.
Our results partly converge with and partly diverge from previous work. Anderson et al. (Reference Anderson, Ort, Kruijne, Meeter and Donk2015) reported that more visually salient regions are fixated sooner, which would predict shorter orienting and processing times for longer, more salient subtitles. By contrast, we found that longer subtitles increased both TFF and processing latency. This apparent contradiction may reflect differences in stimulus context and strategy: in dynamic audiovisual scenes, viewers may be able to evaluate subtitle length parafoveally and delay gaze shifts to longer subtitles because they know those subtitles afford more time for foveal reading, thus producing longer, not shorter, latencies. However, our findings are broadly consistent with research reporting that L2 viewers allocate more attention to subtitles compared to L1 viewers (e.g., Liao et al. Reference Liao, Yu, Kruger and Reichle2022; Szarkowska et al., Reference Szarkowska, Ragni, Szkriba, Orrego-Carmona, Black, Kruger and Krejtz2024a). L2 participants had a higher incidence of refixations, which indicates a difference in early within-subtitle gaze adjustment but does not by itself demonstrate greater reliance on subtitle text. Also, we did not find the predicted faster orienting or processing in L2 readers; instead, L2 viewers were more sensitive to increases in subtitle length at the orienting stage (TFF) and showed distinct micro-saccadic behavior. Taken together, the comparisons suggest that language-driven increases in subtitle attention (Liao et al., Reference Liao, Yu, Kruger and Reichle2022) are evident in saccadic strategies, while the effects of visual salience on latency (Anderson et al., Reference Anderson, Ort, Kruijne, Meeter and Donk2015) interact with the temporal and strategic constraints of viewing moving images.
Altogether, the findings of this study show that subtitle length systematically shapes whether and how subtitles are processed. Very short subtitles are particularly vulnerable to being missed, while longer subtitles slow both attention orientation and processing latency. By contrast, language-related differences are most evident in subtle saccadic correction strategies rather than in broad measures of skipping and latency. Conceptually, this suggests that subtitle reading in dynamic audiovisual contexts is constrained by both temporal/perceptual limits and language-dependent reading strategies. Practically, the results have implications for subtitling guidelines, as subtitle duration and length interact with oculomotor constraints in ways that could undermine readability for certain types of audiences, particularly with less proficient L2 cohorts, as posited above. Implications arise also for literacy and reading development. To date, subtitled video (L1-audio, L1-subtitles) showed promise in L1 literacy, e.g., the case described in Kothari (Reference Kothari2008), who registered improvements in decoding ability and phonemic awareness in low-literate children and adults reading Hindi subtitles on song-based TV programs in India. However, same-language subtitles may no longer be exploitable to their full potential if minimum subtitle durations are too short for beginner readers to successfully map the phonological information of a word in the dialogues to its orthographic form in the subtitles. Finally, from the point of view of L2 language education, minimum subtitle durations that exacerbate ghost subtitle issues mean that teachers would have to be even more careful when selecting subtitled audiovisual texts to use in the classroom so as to ensure that, even in cases where text difficulty is deemed appropriate for the levels of the learners, most subtitles can actually be processed. If this is not possible, then benefits such as increased auditory processing (Winke et al., Reference Winke, Gass and Sydorenko2013), text comprehension (Guichon & McLornan, Reference Guichon and McLornan2008), grammar development (Lee & Révész, Reference Lee and Révész2018), memory retention (Birds & Williams, Reference Bird and Williams2002), and incidental vocabulary learning (e.g., Puimège et al. Reference Puimège, Montero Perez and Peters2023, Wang & Pellicer-Sánchez, Reference Wang and Pellicer-Sánchez2022) resulting from learner use of the subtitles may be partly or completely nullified.
Conclusions
Despite many similarities with the reading of static text, eye movements during subtitle reading differ from print reading in several important ways. Perhaps the most obvious difference is that, in multimodal viewing, the eyes must repeatedly shift between the moving images and the subtitle area in order to integrate meaning across sources. This shift takes time and has a direct bearing on the time available to read the text once the subtitle has appeared. Moreover, once the eyes land on a subtitle, a number of corrective movements are often needed, further delaying the onset of reading before the subtitle disappears. These very early stages of processing are thus critical, yet they remain largely neglected in current research.
The present study addressed this gap by examining subtitle skipping, latency, and initial corrective saccades. The results show that subtitle length plays a central role in determining whether subtitles are attended to and how efficiently they are processed. At the same time, language-group differences were most evident in the microstructure of early corrective behavior rather than in global measures such as skipping or mean latency.
Importantly, our sample comprised proficient English readers in both the L1 and the L2. The findings should therefore not be assumed to generalize to groups with reduced reading proficiency or additional access barriers (e.g., low-proficiency L2 readers, older adults, or deaf or hard-of-hearing viewers). Future work should directly test such groups to establish the generalisability and implications of our findings in terms of accessibility. Furthermore, because the L2 group consisted of L1-Polish speakers only, language-group effects cannot be attributed uniquely to L2 viewing status rather than to characteristics of this specific bilingual population. Future work should include L2 viewers from multiple L1 backgrounds to determine whether the observed patterns generalize across bilingual groups.
Finally, the results point to three clear directions for future research. First, studies should extend beyond the very early stages examined here to later processing. One route would be to model the time course of subtitle reading (linearity vs non-linearity across time) and to consider early eye-movement indices together with later measures such as regression-path duration or total reading times to gain a more complete overview of how subtitle processing unfolds over time. Second, researchers should incorporate word-level measures (e.g., word frequency, predictability, single-fixation probability, and word-level skipping) to determine how lexical properties interact with subtitle length and viewer language to shape both initial orienting and subsequent reading. Third, these psycholinguistic measures of processing (both at the local word level and global subtitle level) should be linked to performance measures such as comprehension accuracy, mnemonic retention, and pre-post-test learning assessments in order to examine the usefulness and usability of subtitles in L2 contexts within and beyond the classroom. Addressing these issues will improve theoretical accounts of reading in multimodal settings, shape subtitle design practices, and inform platform guidelines to enhance accessibility and L2 learning across the lifespan.
Replication package
Replication data for this article can be found at https://osf.io/ybh4s, reference number DOI 10.17605/OSF.IO/YBH4S. Video clips were taken from the Netflix production Don’t Look Up (2021, dir. Adam McKay) with the following approximate timings: Clip 1: 00:02:21–00:08:35; Clip 2: 00:18:00–00:24:06.
Acknowledgements
We would like to acknowledge the contributions of Sharon Black (University of East Anglia), David Orrego-Carmona (Warwick University), and Sonia Szkriba (University of Warsaw) to the study design and data collection.
Competing interests
The authors declare none.
Funding statement
This study was conducted within the WATCH-ME project funded by the National Science Centre OPUS 19 programme 2020/37/B/HS2/00304.
Ethical standard
The study received ethics approval from both the University of East Anglia and the University of Warsaw ethics committees.
Open access
