15.1 Introduction
Our use of language in daily life is rich and complex. Speaking involves the whole body, is embedded in context, and nearly always involves multiple people. Of relevance to the topic of this book, spontaneous spoken language is produced and comprehended on the fly, unfolding continuously over time. These factors contribute to the ongoing emergence of linguistic structure, which is constantly developing to meet our communicational needs. This usage-based approach to the study of language has contributed greatly to the understanding of the cognitive organization of language (Bybee, Reference Bybee, Díaz-Campos and Balasch2023; Bybee and Beckner, Reference Bybee, Beckner, Heine and Narrog2010; Christiansen and Chater, Reference Christiansen and Chater2008; Ellis, Reference Ellis2019; Hopper, Reference Hopper1987; Mithun, Reference Mithun, Givón and Shibatani2009; Mushin and Doehler, Reference Mushin and Doehler2021; Tomasello, Reference Tomasello1998, Reference Tomasello2003). The neural mechanisms supporting the use of language are likely also influenced by these processes (Scott, Reference Scott2019). However, the neural mechanisms of language are often – and perhaps usually – studied by relying on language that is abstracted away from these complexities. In this chapter, we adopt the usage-based perspective and consider the temporal unfolding of spontaneous spoken language and concurrent brain activity. We focus on the prosodic organization of speech into sequences of intonation units (Chafe, Reference Chafe1994; henceforth, IUs), and the ~1 Hz rhythms that they form (Inbar et al., Reference Inbar, Grossman and Landau2020, Reference Inbar, Genzer, Perry, Grossman and Landau2023; Stehwien and Meyer, Reference Stehwien and Meyer2022). Speech prosody is the domain of language that has to do with how pitch, duration, loudness, and other voice qualities unfold over time. Speech prosody plays a central role in communication, and – among its many functions – is used for chunking continuous speech into sequences of prosodic phrases, IUs. Cross-linguistic research suggests that IUs are fundamental to human language and constitute a universal property of language (Chafe, Reference Chafe1994; Himmelmann, Reference Himmelmann2022; Himmelmann et al., Reference Himmelmann, Sandler, Strunk and Unterladstetter2018). In the following, we survey the linguistic literature on IUs, including their formal definition and functional roles in speech (Sections 15.2 and 15.3, respectively). We continue by discussing neuroscientific findings pertaining to IUs, and their relation to low-frequency brain rhythms (Section 15.4). In addition, we review algorithmic tools for IU identification, to facilitate further investigations of IUs and the neural processes associated with them (Box 15.1). We build mainly upon Wallace Chafe’s conception of IUs, yet consider the influence of similar conceptions of prosodic chunking in a variety of research traditions. We aim to provide a comprehensive survey of current research on IUs, and to introduce them within the context of rhythms of speech, language, and the brain.
15.2 Intonation Units: Form
15.2.1 Phonetic-Prosodic Definition of IUs
IUs are defined by a set of phonetic-prosodic characteristics whose identification is an auditory, perceptual matter (Chafe, Reference Chafe1994; Couper-Kuhlen, Reference Couper-Kuhlen1993; Couper-Kuhlen and Barth-Weingarten, Reference Couper-Kuhlen and Barth-Weingarten2011; Cruttenden, Reference Cruttenden1997; Du Bois et al., Reference Du Bois, Cumming, Schuetze-Coburn, Paolino, Du Bois, Cumming, Schuetze-Coburn and Paolino1992; Halliday, Reference Halliday1967; Kibrik, Reference Kibrik2019; Schuetze-Coburn et al., Reference Schuetze-Coburn, Shapley and Weber1991; Shattuck-Hufnagel and Turk, Reference Shattuck-Hufnagel and Turk1996). To identify IUs, one must listen to changes in pitch, duration, and loudness in the unfolding speech. Beginning with pitch, an IU is “a stretch of speech uttered under a single coherent intonation contour” (Du Bois et al., Reference Du Bois, Cumming, Schuetze-Coburn, Paolino, Du Bois, Cumming, Schuetze-Coburn and Paolino1992). Intonation refers to the modulations of pitch in speech, which are controlled by variation in the frequency of vocal fold vibrations. A coherent intonation contour is composed of a unified pitch movement, on the one hand, and a sharp change in pitch level relative to the preceding utterance, on the other. The shape of the pitch movement is often characterized by reference to its nucleus and final part, and it is critical for determining the meaning of an utterance. Fundamental frequency (F0) is the correlate of pitch in the acoustic domain. Although it is an imperfect correlate, it allows to visualize pitch movements and resets. Figure 15.1A presents the F0 time course of an example speech excerpt in which three IUs were identified.
IUs: an example sequence and duration histograms.
An example sequence of three IUs from a Hebrew phone call. From bottom to top: a literal translation of each unit; an X-SAMPA phonemic transcription of Hebrew orthographic words; the audio waveform; the speech envelope; the F0 time series. The highlighted portions in the F0 time series signify actual values measured from voiced segments. Thin lines in between provide an estimation via linear interpolation for voiceless segments and pauses. Speech recording available in Marmorstein et al. (Reference Marmorstein and Matalon2022).
Probability distribution of IU durations in six languages, calculated in 50 ms bins and pooled across multiple speakers. Overlaid are the medians (dashed line, dark gray) and the bootstrapped 95% confidence intervals of the medians (light gray).
Moving to durational characteristics, sequences of IUs include an acceleration–deceleration dynamic of syllable delivery rate. Unit-initial syllables are short and often contracted (a phenomenon termed anacrusis; Cruttenden, Reference Cruttenden1997), while utterance-final syllables are lengthened. Final lengthening is a widespread process in the languages of the world (for recent quantitative demonstrations, see Paschen et al., Reference Paschen, Fuchs and Seifart2022; Seifart et al., Reference Seifart, Strunk and Danielsen2021), and is also considered a highly reliable cue for identifying IU boundaries. The audio waveform and annotations in Figure 15.1A illustrate changes in syllable delivery rate in the example speech excerpt. However, it should be mentioned that the perception of duration is a subjective phenomenon that cannot be fully captured in timing measurements.
In terms of loudness, sequences of IUs tend to include an increase–decrease dynamic in loudness. This dynamic relies on articulatory factors. The degree of contact between articulators is strongest at the onset of a new utterance (Keating et al., Reference Keating, Cho, Fougeron, Hsu, Local, Ogden and Temple2003), and this generates resets in loudness at IU onsets. Additionally, because of the gradual decrease in articulation pressure throughout the IU and the compression of the vocal folds, the vibration of the vocal folds might become irregular and particularly slow near the end of the IU, resulting in a low, raspy voice known as creaky voice (Chafe, Reference Chafe1994).
Pauses in speech often accompany the previously mentioned characteristics. However, they are neither a necessary nor a sufficient characteristic of an IU boundary. IUs may succeed each other without any audible pause in between, and on the other hand, pauses may appear midway through an IU, without a reset in pitch level or a change in syllable delivery rate.
A single IU typically features one accented syllable. Another common configuration is two accented syllables, one primary and the other secondary. Accented syllables are prominent in comparison to surrounding ones due to the pitch movement being centered on the syllable, and they are often louder and/or longer in duration. The accent is usually placed on a lexically stressed syllable, though this is not obligatory. An example exception is the case of contrastive accenting, as in: “I said fifTY not fifTEEN” (Couper-Kuhlen and Barth-Weingarten, Reference Couper-Kuhlen and Barth-Weingarten2011).Footnote 1 See Chapter 35 and Section 6 in general for a complementary view on the rhythm of stressed syllables in speech and its importance in language acquisition.
The phonetic-prosodic definition of IUs provided so far relies on perceptual phenomena, and thus attuned listeners can identify IUs even in unfamiliar languages (Himmelmann et al., Reference Himmelmann, Sandler, Strunk and Unterladstetter2018). The characteristics may be conventionalized and acquire meaning and become language-specific phonological categories. Trivial examples may be found in the association of a specific function with a certain configuration of the coherent pitch contour. For example, it is very common for languages to use particular prosodic contours for questioning. There may be more subtle conventionalized prosodic patterns, for example for strengthening one’s own stance in interaction (Matalon, Reference Matalon, Mauri, Fiorentini and Goria2021).
Linguists have developed different systems for transcribing prosodic phenomena, reflecting a range of research traditions. Three widespread systems in current linguistic research are ToBI (Beckman et al., Reference Beckman, Hirschberg, Shattuck-Hufnagel and Jun2005; Pitrelli et al., Reference Pitrelli, Beckman and Hirschberg1994), GAT (Couper-Kuhlen and Barth-Weingarten, Reference Couper-Kuhlen and Barth-Weingarten2011; Selting et al., Reference Selting, Auer and Barden1998), and DT (Du Bois et al., Reference Du Bois, Cumming, Schuetze-Coburn, Paolino, Du Bois, Cumming, Schuetze-Coburn and Paolino1992), and they all converge on the phonetic-prosodic chunking of speech just described. ToBI (Tones and Break Indices) grew out of the autosegmental-metrical approach to the study of intonation (Ladd, Reference Ladd2008; Pierrehumbert, Reference Pierrehumbert1980). GAT (Gesprächsanalytisches Transkriptionssystem, from German: discourse and conversation-analytic transcription system) was developed as an extension of the transcription style used in conversation analysis (Jefferson, Reference Jefferson and Lerner2004) for the representation of prosody. Similar to GAT, DT (Discourse Transcription) also emerged from the study of everyday conversation. These three systems developed in different research traditions for distinct uses and under varying sets of assumptions. However, they all include explicit marking of IUs, though they call them by different names: intonation units is the term used in DT, GAT uses the term intonation phrases, and ToBI (intermediate) intonational phrases. In the context of transcribing speech, it should be noted that IU boundary annotations tend to be relatively robust between transcribers (e.g., Himmelmann et al., Reference Himmelmann, Sandler, Strunk and Unterladstetter2018; Kibrik, Reference Kibrik2019; Pitrelli et al., Reference Pitrelli, Beckman and Hirschberg1994).
When speech is segmented into IUs, the resulting segmentation reflects an assumption that prosodic phrase boundaries are binary in nature. As speech unfolds, a prosodic boundary either occurs or does not. This assumption may be too simplistic a conception of how speakers actually structure their speech prosodically (Barth-Weingarten, Reference Barth-Weingarten2016; Barth-Weingarten and Ogden, Reference Barth-Weingarten and Ogden2021). Specifically, one can question the utility of a single prosodic cue versus several. Research on talk-in-interaction suggests that granularity may serve a function. For example, the prosodic boundaries between episodes in a narrative (Chafe, Reference Chafe1980) or frames in conversation (Goffmann, Reference Goffman1981) come in different strengths, depending on the amount of reorientation required to process the change.
15.2.2 Temporal Structure of IUs
Previous studies characterized the length of IUs using different measurement styles. One method measures the number of words per IU (Chafe, Reference Chafe1994; Himmelmann et al., Reference Himmelmann, Sandler, Strunk and Unterladstetter2018; Pawley and Syder, Reference Pawley, Syder and Riggenbach2000). We present the counts aggregated from these studies in Table 15.1. However, as we have demonstrated elsewhere (see Box 2 in Inbar et al., Reference Inbar, Grossman and Landau2020), what constitutes a word varies greatly across languages (Haspelmath, Reference Haspelmath2011), and so it is difficult to infer the temporal structure of IUs from word counts. Other studies measured the number of syllables per IU (see Table 15.1). Syllable structure also varies across languages, yet unlike the case for words, it is possible to find a common formal characterization for them: Syllables evolve around a vocalic nucleus (e.g., Blevins, Reference Blevins2006). Thus, it is possible to count and measure the durations of syllables in a cross-linguistically sensible way (Greenberg, Reference Greenberg2022; Greenberg et al., Reference Greenberg, Carvey, Hitchcock and Chang2003) and obtain an indirect estimation of IU length. Recent studies found that languages consistently differ in syllable production rate (Coupé et al., Reference Coupé, Oh, Dediu and Pellegrino2019; Pellegrino et al., Reference Pellegrino, Coupé and Marisco2011). Currently, it is unclear if this difference leads to differences in IU rates across languages, or if languages differ in the syllable counts per IU. Finally, it should be noted that an estimation of IU length does not capture information about the temporal structure of IU sequences.
| Language | Word count | Syllable count | IU type (see Section 15.3) | Reference |
|---|---|---|---|---|
| English | 4.84 | Substantive, e.g., “we had such a nice day” | Chafe (Reference Chafe1994) | |
| 1.36 | Regulatory, e.g., “and” “well” “umm” | |||
| Seneca (Iroquoian, east USA and Canada) | 2 | Any | ||
| German | 5.20 | Any | Himmelmann et al. (Reference Himmelmann, Sandler, Strunk and Unterladstetter2018) | |
| Papuan Malay | 4.08 | Any | ||
| Wooi (Austronesian, West Papua) | 3.63 | Any | ||
| Yali (Papuan highland, West Papua) | 3.37 | Any | ||
| Modern Hebrew | 2.18 | Any | Inbar et al. (Reference Inbar, Genzer, Perry, Grossman and Landau2023) | |
| English | 7–10 | Excluding IUs “consisting of a connective or an exclamation (e.g., Alors, ‘Therefore’, in French)” | Jun, Reference Jun and Jun2005 | |
| French | 7–10 | |||
| Japanese | 7–10 | |||
| Korean | 12–15 |
In a recent study, we provided a quantitative characterization of the temporal structure of IU sequences (Figure 15.1B; see also Figure 2 in Inbar et al., Reference Inbar, Grossman and Landau2020). We analyzed audio recordings of spontaneous speech in six languages from different families and from different parts of the world. We used IUs as times of interest for analyzing periodicities in the speech envelope. The speech envelope (see Figure 15.1A) is a representation of amplitude fluctuations in speech, and it therefore presumably captures two of the cues for IU boundaries: (1) resets in loudness, and (2) the acceleration–deceleration dynamic of syllable delivery, because one of the sources for the amplitude peaks measured in the speech envelope is the succession of syllable nuclei (see Section 2 for further discussion of amplitude fluctuations in speech). We found that ~1 Hz fluctuations in the speech envelope aligned consistently with IU onset times, and consequently delineated the rate of the unfolding of IUs at a rate of approximately one per second. This finding held in each of the six languages that we studied, despite substantial differences between them in grammatical, areal, genealogical, and socio-cultural parameters. Stehwien and Meyer (Reference Stehwien and Meyer2022) replicated these results in an additional language, finding that sequences of 2–5 IUs in German are periodic at 0.8–1.6 Hz. This study used a different statistical framework and radio news broadcasts rather than recordings of spontaneous speech, providing further validity and robustness to the finding. Together, these studies (Inbar et al., Reference Inbar, Grossman and Landau2020; Stehwien and Meyer, Reference Stehwien and Meyer2022) corroborate early reports on the temporal structure of IU sequences of spoken American English (Chafe, Reference Chafe and Tomlin1987; Couper-Kuhlen, Reference Couper-Kuhlen1993) with quantitative evidence from a variety of languages.
This subsection described the temporal structure of sequences of IUs, namely, that one IU succeeds another at the rate of approximately one IU per second. Although the rhythm we characterize is not an isochronous beat, to which one can dance, tap, or nod (Cummins, Reference Cummins2009), nuanced research can demonstrate cases where the approximate rhythmic structure does, in fact, shape behavior. To conclude this subsection, we would like to present evidence that in local contexts, sequences of IUs, or rather sequences of accentuated syllables within IUs, do happen to give rise to rhythmicity in the sense of affording synchronization. Linguists have studied such rhythmic structures in talk-in-interaction (Auer et al., Reference Auer, Couper-Kuhlen and Müller1999; Couper-Kuhlen, Reference Couper-Kuhlen1993; Marmorstein and Matalon, Reference Marmorstein and Matalon2022). In their analyses of stretches of spontaneous spoken language, they identify rhythmic structures that are composed of at least three accentuated syllables (e.g., in three IUs) that are evenly spaced in time. It is possible to tap along to these structures, and there is further evidence that interlocutors perceive this rhythmicity and act on it during interaction. Specifically, when interlocutors take turns at talk, rhythmic structure can be constructed also across their turns, giving rise to rhythmic integration (Auer et al., Reference Auer, Couper-Kuhlen and Müller1999). When one interlocutor integrates their speech into a rhythm set by another at a point of speaker change, this promotes intact interactional exchange. However, when the speech is rhythmically off, interlocutors “engage in supplementary inferencing to account for the departure from expectation” (Auer et al., Reference Auer, Couper-Kuhlen and Müller1999, p. 70). In other words, the talk following a potential point of integration across turns provides evidence for the rhythmicity. The phenomenon of rhythmic integration highlights the importance of studying spontaneous speech in interactional contexts, for finding rhythm in language.
The identification of IUs has been for a long time an auditory, perceptual matter. The task of manual speech segmentation is laborious and time-consuming, and cannot keep up with the vast amounts of speech recordings. This limits the breadth of investigations pertaining to IUs, in terms of the languages studied and the amount of evidence in each. Automatic tools to speed up the manual process are much called for, yet the development of such tools relies on the unresolved relationship between acoustics and auditory perception. The reader is referred to Biron et al. (Reference Biron, Baum and Freche2021) for an overview of the existing approaches to boundary detection, as well as a discussion of their strengths and shortcomings.
Here we describe two applications that managed to provide segmentations of speech into IUs at an accuracy level above chance classification in large corpora of American English, Biron et al. (Reference Biron, Baum and Freche2021) and Suni et al. (Reference Suni, Šimko, Aalto and Vainio2017). The two applications operationalize the cues for IU boundaries, finding signals and thresholds that correspond to the perception of a boundary. In both cases, the applications relied on the acceleration–deceleration dynamic of speech delivery rate. This dynamic is often considered the most salient characteristic of IU boundaries. In the application described by Biron et al. (Reference Biron, Baum and Freche2021), an instantaneous measure of speech rate was derived from time-stamped phone annotations and pauses. Speech rate calculated this way was found to drop by approximately twofold at the end of IUs in conversational American English, allowing for efficient automatic IU boundary detection. For the calculation of the instantaneous speech rate, this method requires phone timings, that is, time stamps at the level of individual speech sounds. Identifying and annotating phone timings is itself a laborious task, and is often achieved by automatic forced-alignment tools, with suboptimal results. Code streamlining the application presented in Biron et al. (Reference Biron, Baum and Freche2021) is unpublished but available.
The Wavelet Prosody Toolkit (Suni et al., Reference Suni, Šimko, Aalto and Vainio2017) is an application for the hierarchical representation and estimation of prosody. This algorithm derives F0 and intensity information from the speech audio signal, and duration information from time-stamped word annotations. Thus, the way that the acceleration–deceleration dynamic is captured differs from that of Biron et al. (Reference Biron, Baum and Freche2021). Speech rate is operationalized based on the word level rather than the phone level, and is additionally captured in the speech intensity signal, which gives an indirect measure of the syllable delivery rate. In addition, the application of Suni et al. relies on resets in loudness at IU boundaries, as captured in the speech intensity signal, as well as resets in pitch level at IU boundaries, as captured in the F0 signal. Together, these signals are subjected to an analysis that extracts from them a measure of discontinuity – the boundary strength score. Words that are associated with a strong boundary strength score are classified as ending an IU, thereby providing the segmentation. Suni et al. used a threshold for classification that optimized the performance in their training set relative to manual annotations, and more work is required to survey the stability of thresholds across speakers and languages. Note that this method requires word timings rather than phone timings. Thanks to ongoing developments in automatic speech recognition techniques, word annotations are becoming easier to obtain automatically and in ever more languages. In the context of language neuroscience, the acoustic boundary strength measure developed by Suni et al. (Reference Suni, Šimko, Aalto and Vainio2017) has already been utilized in several recent studies (Anurova et al., Reference Anurova, Vetchinnikova and Dobrego2022; Degano et al., Reference Degano, Donhauser, Gwilliams, Merlo and Golestani2023; Inbar et al., Reference Inbar, Genzer, Perry, Grossman and Landau2023; Vetchinnikova et al., Reference Vetchinnikova, Konina, Williams, Mikušová and Mauranen2023). These studies highlight the importance of using ecological stimuli in the study of language and its neural mechanisms, and demonstrate the gains in modeling the temporally unfolding prosodic structure. Algorithmic tools for IU identification facilitate the incorporation of IUs in experimental and quantitative research, and provide the stepping stone to study the construct of the IU itself on a large scale and in ever more languages.
15.3 Intonation Units: Function
The prosodic phrasing described in Section 15.2 provides a systematic and temporally well-defined perceptual segmentation to ongoing naturalistic speech. Chafe (Reference Chafe and Tomlin1987, Reference Chafe1994) termed the segmentation under discussion intonation units. We adopt this term rather than others because Chafe foregrounded the role of the prosodic segmentation of speech in communication. In the current section we describe the role of IUs in pacing the flow of information in the discourse, together with additional characterizations of their function. It is this aspect of IUs that has led Chafe and others working on language in day-to-day discourse to view IUs as a window onto dynamic cognitive processes in language (e.g., Chafe, Reference Chafe1994, Reference Chafe2018; Du Bois, Reference Du Bois1987; Kibrik, Reference Kibrik2019; Mithun, Reference Mithun, Raso and Mello2014; Park, Reference Park2002; Wahl, Reference Wahl2015).
In conversation, speakers build a common ground between them step by step. For example, when developing a narrative, speakers introduce the setting, participants, and the course of events in a gradual manner. Ideas that indeed call for introduction relative to the preceding discourse are produced one at a time, in separate IUs. This presumably derives from cognitive constraints on the amount of new information speakers can produce and comprehend. Such constraints may relate to attentional capacity and memory limitations (e.g., Christiansen and Chater, Reference Christiansen and Chater2016; Simpson and del Prado Martín, Reference Simpson and del Prado Martín2015). Chafe (Reference Chafe1994) termed this the one-new-idea constraint. IUs may include ideas that are not new – specifically, ideas that have been previously introduced in the discourse, or that have been evoked by an introduced schema (Du Bois, Reference Du Bois1987). These may appear beside new ideas in a single IU, or in an IU of their own. However, IUs appear to express no more than one new idea at a time.
In discourse studies, new ideas are identified based on in-depth qualitative investigations of the discourse transcription. This type of investigation does not lend itself easily to analyses of large amounts of data, and so the field is lacking quantitative evidence for the one-new-idea-per-IU constraint across many languages and contexts. How might one operationalize newness of ideas in spontaneous discourse? In recent years it has become popular to estimate the amount of information that individual words convey using probabilistic language models (e.g., Armeni et al., Reference Armeni, Willems and Frank2017; Brodbeck et al., Reference Brodbeck, Bhattasali and Cruz Heredia2022; Broderick et al., Reference Broderick, Anderson, Di Liberto, Crosse and Lalor2018; Frank et al., Reference Frank, Otten, Galli and Vigliocco2015; Goldstein et al., Reference Goldstein, Zada and Buchnik2022; Hale, Reference Hale2001; Levy, Reference Levy2008; Nelson et al., Reference Nelson, El Karoui and Giber2017; Shannon, Reference Shannon1948; Xu and Reitter, Reference Xu and Reitter2018). The field of natural language processing is booming with development in this area, offering constant improvement in the ability to capture context in its full complexity. With better models of context, the estimation of informativeness is expected to be more felicitous. Such informativeness estimates, when considered relative to the prosodic segmentation of speech into IUs, may provide quantitative evidence for the one-new-idea constraint. Cross-linguistic research on phone and syllable rate suggests that language production exhibits a trade-off between production rate and informativeness, as operationalized by similar probabilistic models (Coupé et al., Reference Coupé, Oh, Dediu and Pellegrino2019; Pellegrino et al., Reference Pellegrino, Coupé and Marisco2011; Pimentel et al., Reference Pimentel, Meister and Salesky2021). This trade-off is hypothesized to stem from a communicative need to spread redundancy evenly across the speech signal (smooth signal redundancy and uniform information density hypotheses; Aylett and Turk, Reference Aylett and Turk2004; Jaeger, Reference Jaeger2010). It is possible that prosodic chunking answers a similar need (as suggested in Turk, Reference Turk2010), and that the one-new-idea constraint described in qualitative linguistic research could be understood within the broader principle of uniform information density.
Chafe proposed to distinguish between three types of IUs, namely, substantive, regulatory, and fragmentary IUs (for an illustration of the three types, see Chafe, Reference Chafe1994, p. 63). Starting with the latter, some IUs stop in the middle of production. These are termed fragmentary IUs, and are identified based on their truncated prosodic nature. IUs that are not truncated but rather produced in full may be subcategorized as regulatory or substantive. Regulatory IUs are those that have regulatory functions, such as organizing the discourse structure, addressing interpersonal matters between interlocutors, and expressing the speaker’s mental processes. As noted by Chafe, regulatory IUs often coincide with the linguistic devices called discourse markers (Maschler and Schiffrin, Reference Maschler, Schiffrin, Tannen, Hamilton and Schiffrin2015), and similarly pertain to the process of using language itself. Conversely, substantive IUs are those that pertain to the extra-lingual world, conveying ideas of events, states, or referents.
To characterize the function of IUs, one might consider their syntactic makeup. As just mentioned, substantive IUs convey ideas of events, states, or referents. Syntactically, ideas of events and states are typically produced in the form of a clause, with a predicate and all the complements and modifiers related to it. And indeed, the most common form of an IU is a clause (Chafe, Reference Chafe1994; see also Givón, Reference Givón2018; Pawley and Syder, Reference Pawley, Syder and Riggenbach2000). This has been demonstrated in a variety of studies in several languages, including English, Japanese, Mandarin Chinese, German, Russian, Sasak (Austronesian, Indonesia), Dolakhae Newar (Sino-Tibetan, Nepal), Wardaman (Yangmanic, Australia), and Korean (for reviews, see Croft, Reference Croft1995, Reference Croft2007; Kibrik, Reference Kibrik2019; Park, Reference Park2002). IUs are nearly always full syntactic units of other types if not clauses (e.g., noun phrases, adverbs), containing a grammatical element with all its complements. Croft suggested that such units are “relatively highly integrated … stored as wholes in the mind and produced as single ‘bursts’ of articulation, that is, as a single intonation unit” (Croft, Reference Croft2007, p. 5).
Finally, besides reflecting cognitive constraints on information flow, IUs are key units in the construction of interaction in an additional way. Transitions between interlocutors in conversation are smooth and swift, with little overlap and very short lags between turns (Levinson, Reference Levinson2016; Sacks et al., Reference Sacks, Schegloff and Jefferson1974; Schegloff, Reference Schegloff2007). Interlocutors coordinate these transitions, and must rely on different cues for the task, that is, on different projections of upcoming completion. In interactional linguistics, projection means that an earlier part of a structure foreshadows its later trajectory, thus making its completion predictable (Auer, Reference Auer2005; Couper-Kuhlen and Selting, Reference Couper-Kuhlen, Selting, Couper-Kuhlen and Selting2017). Linguistic structures at various levels have recognizable trajectories, and so projections may be content-based, syntactic, phonological, prosodic, as in the case of IUs, and more. Prosodic projection via the formal characteristics of IUs – as described in Section 15.2 – has been shown to serve a crucial role in timing turn transitions (Bögels and Torreira, Reference Bögels and Torreira2015, Reference Bögels and Torreira2021; Ford and Thompson, Reference Ford, Thompson, Ochs, Schegloff and Thompson1996; Gravano and Hirschberg, Reference Gravano and Hirschberg2011; Selting, Reference Selting1996) and in temporal predictions in language processing more generally (Lamekina and Meyer, Reference Lamekina and Meyer2023).
15.4 Intonation Units: Cognitive Neuroscience
Spoken language is prosodically structured in sequences of IUs that succeed each other at a rate of approximately one unit per second. Usage-based linguistic research suggests that these phonetically defined units provide a window onto higher-level representations that are at play during language use. For example, as discussed in Section 15.3, IUs appear to pace new information in the discourse. They also allow speakers to anticipate turn changes in conversation. Against this backdrop, we may ask what cognitive functions and underlying neural mechanisms are associated with the processing of IUs. In this section we survey a study that found a neural response associated with IU closure. This response was found to relate to low-frequency brain rhythms as characterized using electrophysiological recordings. We discuss the neural response to IUs in light of previous neuroscientific studies on speech comprehension, including classical event-related potential (ERP) literature as well as literature on brain rhythms and cognition.
15.4.1 Neural Signatures
In a recent study, we measured the electroencephalography (EEG) of participants who listened to different speakers recounting an emotional life event in Hebrew (Genzer et al., Reference Genzer, Ong, Zaki and Perry2022; Inbar et al., Reference Inbar, Genzer, Perry, Grossman and Landau2023). We analyzed the stimuli into IUs based on close auditory listening and categorized each word as either ending an IU or not. Additionally, we quantified an acoustically based measure of prosodic boundary strength at each word (see Box 15.1). We modeled the neural response at each EEG channel and time point relative to word offset. We found that the EEG response to words that end an IU differs from the response to words that do not end an IU, even when equating acoustic boundary strength. The EEG response at IU closure includes a negative deflection at right-anterior electrodes, starting shortly before the end of the final word in the IU and lasting circa 200 ms (Figure 15.2A). IU closure is further characterized by a centroparietal positive deflection between 150 and 500 ms after the final word in the IU (Figure 15.2B). In addition, we found that stronger acoustic boundaries elicit a larger anterior negativity between 100 and 400 ms (Figure 15.2C). Thus, when listening to spontaneous speech, the neural system is attuned to IUs. The response to IUs extends beyond a response to mere acoustics and captures processing at higher levels of representation that presumably unfolds in sequences of IUs.
IUs: neural response.
ERP traces in response to words that end an IU and other words with comparable acoustic boundary strength (dark and light gray, respectively), illustrating the right-anterior negative cluster. The traces show the grand average over the EEG electrodes highlighted in the inset topography. Shaded ribbons correspond to ±1 SEM. The horizontal bar marks time points of significant difference between conditions (corrected for multiple comparisons).
ERP traces in response to words that end an IU and other words with comparable acoustic boundary strength (dark and light gray, respectively), illustrating the centroparietal positive cluster. The traces show the grand average over the different sets of EEG electrodes highlighted in the inset topography. Shaded ribbons correspond to ±1 SEM. The horizontal bar marks time points of significant difference between conditions (corrected for multiple comparisons).
ERP traces in response to words that end an IU at different levels of acoustic boundary strength, illustrating the anterior negative cluster. Four different levels are presented, corresponding to quartiles of boundary strength scores within words that end an IU. The darker the gray, the stronger the acoustic boundary. The traces show the grand average over the EEG electrodes highlighted in the inset topography. Shaded ribbons correspond to ±1 SEM. The horizontal bar marks time points of significant difference between conditions (corrected for multiple comparisons).
Group average partial correlation maps of the unique predictive accuracy of IU closure (left) and acoustic boundary strength (right), when predicting left-out EEG data in the delta band. Highlighted electrodes denote electrodes in which the predictive accuracy was significantly higher than chance (corrected for multiple comparisons).
Within the ERP literature, previous work has identified a component that resembles the response described in the previous paragraph, the closure positive shift (CPS) (Bögels et al., Reference Bögels, Schriefers, Vonk and Chwilla2011; Steinhauer et al., Reference Steinhauer, Alter and Friederici1999). The CPS was found in response to prosodic phrase boundaries in several languages other than Hebrew, including German, Dutch, English, Swedish, Japanese, Chinese, and Korean. See a detailed account of the CPS in Chapters 17 and 18. Prosodic boundaries in these studies rely on cues that are nearly identical to those defining IUs. Importantly, however, these studies always used isolated, constructed sentences, and so, to the best of our knowledge, our analysis was the first to characterize a CPS-like response to spontaneous speech during naturalistic listening conditions. Furthermore, our modeling approach allowed to address a debate in the CPS literature considering the extent to which the CPS reflects a linguistic structuring of the input beyond the bottom-up response to acoustic boundary cues (Itzhak et al., Reference Itzhak, Pauker, Drury, Baum and Steinhauer2010; Kerkhofs et al., Reference Kerkhofs, Vonk, Schriefers and Chwilla2007; Pannekamp et al., Reference Pannekamp, Toepel, Alter, Hahne and Friederici2005; Pauker, Reference Pauker2013; Steinhauer, Reference Steinhauer2003). Our results suggested what appear to be two different components within the classical CPS. The early- and right-anterior negativity and the centroparietal positivity do not depend on acoustic boundary strength, while the late-anterior negativity is dependent on acoustic boundary strength. This neat result was made possible by combining linguistic theory that focuses on language use with a modeling approach that considers trial-level dynamics among multiple factors.
15.4.2 Brain Rhythms
Research in recent decades has highlighted the relation between temporal structure in speech and temporal structure in brain activity. Understanding this relationship and its role for language has been an influential research program and is one of the goals of the current book. For a detailed presentation of temporal structure in brain activity, see Chapter 3. This line of research suggests that neural activity tracks speech moment by moment (Giraud and Poeppel, Reference Giraud and Poeppel2012; Gross et al., Reference Gross, Hoogenboom and Thut2013; Obleser and Kayser, Reference Obleser and Kayser2019). Neural speech tracking refers to any significant relationship between brain activity and a speech stimulus. Neural speech tracking has been found in different timescales, among them, the theta band (4–8 Hz). This temporal scale is thought to reflect neural tracking of syllables, which across languages tend to have a similar temporal structure: four–eight syllables per second (Chandrasekaran et al., Reference Chandrasekaran, Trubanova, Stillittano, Caplier and Ghazanfar2009; Ding et al., Reference Ding, Patel and Chen2017; Greenberg et al., Reference Greenberg, Carvey, Hitchcock and Chang2003; Pellegrino et al., Reference Pellegrino, Coupé and Marisco2011; cf. MacIntyre et al., Reference MacIntyre, Cai and Scott2022). Neural speech tracking has also been found in the delta band (1–2 Hz); however, in this band there was, until recently, less understanding as to what speech components are being tracked (see detailed discussions in Chapters 17 and 21). From a continuous perspective, since sequences of IUs in spontaneous speech give rise to rhythms of ~1 Hz, the time course of neural responses to IUs is expected to give rise to delta band neural speech tracking. In Inbar et al. (Reference Inbar, Genzer, Perry, Grossman and Landau2023), we showed precisely this. IU-related activity captures previously described delta band neural activity (Figure 15.2D). Specifically, IU-related potentials were able to accurately predict unseen continuous EEG responses to our stimuli in the delta band. This finding was made possible by our analytic approach: We relied on regression ERPs and then used the stimulus parameters and the fitted potentials to predict continuous EEG signals.
Note that Meyer et al. (Reference Meyer, Henry, Gaston, Schmuck and Friederici2016, Reference Meyer, Sun and Martin2020) hypothesized that there may be a link between low-frequency neural speech tracking and the CPS. As discussed briefly in Section 15.4.1, the CPS literature debated to what extent the component reflects a linguistic structuring of the input beyond the bottom-up response to acoustic boundary cues. This debate is also at the crux of the uncertainty relative to speech tracking in the delta band. Thanks to our modeling approach, in which the EEG was regressed against multiple predictors, low-frequency speech tracking could be probed relative to individual predictors while accounting for activity explained away by other predictors. Acoustic boundary cues were found to uniquely predict delta band neural activity in one cluster (Figure 15.2D, right), whereas IU-closure status uniquely predicted delta band neural activity in a separate cluster (Figure 15.2D, left). We also studied the unique contribution of one specific higher-level linguistic representation to delta band neural activity, namely clause closure (see Figure 6 in Inbar et al., Reference Inbar, Genzer, Perry, Grossman and Landau2023). Clause closure also predicted unique delta band neural activity, yet to a lesser extent than IU closure and in a smaller cluster.
The current section has dealt exclusively with IUs in speech comprehension. We discussed IUs in speech comprehension as perceptual units relevant also to higher-level cognitive processes. One might expect that focusing on speech production will be no less illuminating regarding the cognitive processes and underlying neural mechanisms associated with IUs. This topic is beyond the scope of the current chapter, yet we would like to raise a question for future research. Speech production involves an ensemble of motor components, and some have suggested that speech timing might emerge from dynamical interactions between these motor components (Turk and Shattuck-Hufnagel, Reference Turk and Shattuck-Hufnagel2014). Are the motoric facets of IU production the sole contributors of their temporal structure? Conversely, perhaps we might identify additional, higher-level cognitive processes at play, such as the process of selecting what to say in the first place, involving both attention and memory (see Chapter 18).
15.5 Conclusion
This chapter provides a step forward in identifying a promising theoretical construct in the study of cognition, especially within the context of rhythms of language and the brain. We focus on the prosodic organization of spontaneous spoken language into intonation units (IUs). Prosodic chunking into IUs is fundamental to human language. We have presented the phonetic-prosodic definition of IUs and current research regarding their temporal structure. We have discussed the roles of IUs in pacing the flow of information in the unfolding discourse, as revealed in studies of everyday conversation. We have surveyed open questions in the neuroscientific literature that recently benefited from considering IUs in speech: questions pertaining to the nature of the CPS and delta band speech tracking. Finally, we have reviewed algorithmic tools for IU identification to facilitate further investigations relying on this construct. Our approach highlights the importance of studying language mechanisms from the perspective of their use, and applying advanced analytical methodologies for this task.
Summary
Spontaneous spoken language is structured in time at the level of prosodic phrasing, a presumably universal property of language. Linguistic research suggests that this structure offers a window onto dynamic cognitive processes during communication. Neural speech tracking in the delta band is predictable based on the prosodic organization of speech into IUs.
Implications
Considering language in its natural occurrence, that is, communication, has proven crucial for describing rhythm in language and the brain. Future research on the cognitive neuroscience of language will benefit from considering the complexities and richness of the way we use language.
Gains
The chapter introduced key points from linguistic research on IUs, and offered a usage-based approach to the neuroscientific study of language. The chapter further surveyed automatic applications for arriving at a prosodic segmentation of speech. These tools might facilitate investigations on prosodic phrasing and its interface with other domains of language and cognition in ecological settings.
Open access