4.1 Introduction
Seated on a train waiting to leave the station, you notice movement out of the corner of your eye and think to yourself, “Finally, we’re on our way!” However, as the train on the adjacent track continues to move, you realize that your own train is still stationary. This phenomenon, known as vection, is an illusion of self-motion thought to arise due to conflicting signals from the visual system and the vestibular system (Dichgans and Brandt, Reference Dichgans, Brandt, Held, Leibowitz and Teuber1978). The vestibular system includes a sophisticated set of organs in the inner ear responsible for detecting changes in self-motion and orientation in space. Historically, these organs and their connections across multiple cortical and subcortical brain areas have received limited attention in cognitive neuroscience, as they have been assumed to be predominantly relevant for functions such as postural control (vestibulocollic reflexes), gaze stabilization (vestibular-ocular reflexes), and autonomic regulation (Angelaki and Cullen, Reference Angelaki and Cullen2008).
However, a growing body of evidence suggests vestibular information contributes to a variety of cognitive processes (Mast and Ellis, Reference Mast and Ellis2015; Harris, Reference Harris2020). Following this trend, we suggest that the computational properties of the vestibular system make it useful for processing the prosodic elements of speech. We focus on a phenomenon we refer to as multimodal prosody: aspects of the prosodic signal in speech that are reliably paired with visual information from the speaker’s body. Dynamic changes of suprasegmental components of speech, including pitch and amplitude, are mirrored in co-speech gestures (Wagner et al., Reference Wagner, Malisz and Kopp2014). In this chapter we present the sensorimotor account of multimodal prosody (SAMP) that links the vestibular system to multimodal prosody and motivates novel hypotheses regarding speech-gesture integration and how attention is structured during language exchange.
We begin below with a brief introduction to the anatomy and physiology of the vestibular system. In Section 4.3, we review evidence showing that the vestibular system contributes to dynamic attentional control, timekeeping functions, and musical rhythm perception. In Section 4.4, we discuss a potential role for the vestibular system in the perception of rhythm in speech. Because human vocalization provides an above-threshold mechanical stimulus for the vestibular end organs, we propose that the brain treats speech as an acoustic-vestibular stimulus, and thus a form of self-motion. Moreover, because speech production is both accompanied and constrained by movements of the body, vestibular mechanisms are likely involved in encoding and decoding prosodic features of multimodal discourse. We present our sensorimotor account of multimodal prosody (SAMP) in Section 4.5, and conclude in Section 4.6 by highlighting the potential clinical relevance of the proposal for populations with language and/or rhythmic processing difficulties.
4.2 The Vestibular System: New Perspectives
For the purposes of this chapter, discussion of the vestibular system will generally focus on the information processing capacity of the otolith organs, which are constantly stimulated by downward gravitational forces, and encode linear translations of the head. We begin by highlighting basic anatomical and physiological principles of the peripheral vestibular function, and how vestibular information is integrated with exteroceptive and interoceptive information from early stages in the cortical hierarchy. Following this, we briefly discuss the predictive coding scheme that characterizes vestibular information processing in the brain.
4.2.1 Anatomy, Physiology, and Functional Principles
Within each ear, three semicircular canals encode rotational movements of the head, and two otoliths detect gravito-inertial acceleration, that is, the vector sum of linear head translations and gravity (see Figure 4.1). The otolith organs contain particles of calcium carbonate suspended in a membrane that flows according to inertial changes in head tilt or motion. Head motion and gravitational forces cause mechanical displacement of hair cell stereocilia and generate potentials that propagate to the vestibular nuclei in the medulla, where information about head motion is relayed to various regions of the cortex, both directly from the brainstem as well as through the thalamus (Kirsch et al., Reference Kirsch, Keeser and Hergenroeder2016; Cullen, Reference Cullen2019). Vestibular mechanoreceptors are highly sensitive to physical perturbations; for example, afferent nerves can ignite in response to hair cell deflections of less than half a nanometer (Curthoys et al., Reference Curthoys, Grant and Burgess2018).
Vestibular anatomy.
During head rotations, endolymph fluid lags behind angular rotation of the head due to inertia, and its displacement bends the cilia of the hair cells in one of the three canals, anterior, lateral, and posterior.
Within the macula of the utricle and saccule, which encode horizontal and vertical accelerations of the head, respectively, the otolith crystals move in response to gravity and head motion, bending stereocilia of the hair cells and triggering an afferent response.
The signal transduced by the otoliths is called gravito-inertial acceleration, which is the vector sum of forces due to gravity and inertial motion of the head. These have the same physical consequences on the otolith organs, such that tilting the head backward and accelerating forward with the head upright results in the same signal to the brain via cranial nerve VIII.
Figure 4.1C: Long description
Part C depicts the mechanics of how the otolith organs detect linear acceleration and head tilt. Two intersecting lines depict gravito-inertial acceleration. A figure of a human head labeled backward tilt. The intersecting lines marked on the same head indicate a block of hair cells, followed by an equal symbol, a figure of a human head labeled forward translation. The intersecting lines marked on the same head indicate a block of hair cells. A coordinate system. The Y-axis is vertical, the Z-axis is horizontal, and the X-axis is perpendicular to both. A vector pointing to the right in the Y-Z plane represents the inertial force due to linear translation. A vector pointing downwards in the Y-axis represents the force of gravity. The X-axis represents the gravito-inertial acceleration.
Vestibular afferent information is extensively connected to the cerebellum, basal ganglia, and other subcortical structures important for movement control (Stiles and Smith, Reference Stiles and Smith2015; Rühl et al., Reference Rühl, Kimmel, Ertl, Conrad and Zu Eulenburg2023). In humans, vestibular-sensitive regions of the cortex are very diverse and have been identified in areas near the temporoparietal junction, including the supramarginal and angular gyri, and the posterior superior temporal gyrus, parietal operculum, and posterior insular cortex (see Lopez et al., Reference Lopez, Blanke and Mast2012, for a review). Notably, there is considerable overlap between vestibular-sensitive regions of the cortex and traditional areas of the language network as defined by Hickok and Poeppel (Reference Hickok and Poeppel2007), including the superior planum temporale (SPT), BA 21, BA 22 (Wernicke’s area), and BA 44/45 (Broca’s area) (Gattie et al., Reference Gattie, Lieven and Kluk2021).
The vestibular system is unusual for several reasons – one of which is its inherent multimodal nature; that is, all neurons in the brain that respond to vestibular stimulation are also activated by some other source of sensory stimulation (Dieterich and Brandt, Reference Dieterich and Brandt2015). Vestibular afferent information becomes integrated with visual, somatosensory, and proprioceptive signals at very early processing stages in the brainstem (Cullen, Reference Cullen2019). In contrast to other afferent systems in the central nervous system (CNS) (e.g., Heschl’s gyrus, V1, S1), it is generally accepted that no true “primary” vestibular cortex exists. An interesting corollary of its multisensory nature is that the vestibular system is important for regulating afferent gain across other sensory modalities, including vision (e.g., Voros et al., Reference Voros, Sherman and Rise2021), somatosensation (Ferrè et al., Reference Ferrè, Walther and Haggard2015), proprioception (Ponzo et al., Reference Ponzo, Kirsch, Fotopoulou and Jenkinson2019), and even nociception (Ferrè et al., Reference Ferrè, Bottini, Iannetti and Haggard2013).
Cell populations in the peripheral organs of the vestibular system are categorized by their firing regularity at resting state. Irregular afferents demonstrate relatively high resting-state firing variability, are particularly sensitive to changes in head motion by their use of spike timing precision, and help ensure maintenance of posture during changes in self-motion. By contrast, regular afferents exhibit low firing-rate variability during rest, and become rapidly integrated with eye movement signals to mediate reflex-arcs, such as the vestibulo-ocular reflex, that improve gaze fixation during head motion (Sadeghi et al., Reference Sadeghi, Chacron, Taylor and Cullen2007). Besides the time course of head motion, regular afferents’ use of a rate-coding strategy makes them well suited to encode temporal information more generally (Jamali et al., Reference Jamali, Carriot, Chacron and Cullen2019).
4.2.2 Modeling Self-Location
The vestibular system makes critical contributions to dynamic representations of the movement and location of the organism, providing continuous signals about angular and linear acceleration of the head by the semicircular canals and otoliths, respectively (Laurens and Droulez, Reference Laurens and Droulez2007). Information from other sensory modalities, such as vision and proprioception, also constrains estimates of these parameters. In fact, some inherent limitations of the vestibular end organs mean that multisensory integration is imperative for successful self-motion encoding in the brain (Laurens and Angelaki, Reference Laurens and Angelaki2011). This capacity of the vestibular system for multisensory integration makes it well suited for coding correlated streams of information from different modalities, such as the acoustic properties of speech and movements associated with accompanying gestures.
Further, the vestibular system has been shown to issue predictions about the consequences of bodily movements, that is, how motor commands are expected to impact signals from the semicircular canals and otoliths (Laurens and Angelaki, Reference Laurens and Angelaki2017). Indeed, the earliest level of convergence between the vestibular periphery and the nervous system involves a comparison between the expected consequences of self-motion against the actual dynamic vestibular afferent signal (Dale and Cullen, Reference Dale and Cullen2019). In order to predict afferent information transmitted by the vestibular periphery, the vestibular system needs to internally represent features of the environment likely to influence how the head registers changes in forces, including gravity and biomechanical properties of the organism’s own body (e.g., Laurens et al., Reference Laurens, Meng and Angelaki2013; Martin et al., Reference Martin, Lapierre, Haché, Lucien and Green2021).
4.3 Temporal Expectations in the Body
Besides its accepted role in maintaining balance, the vestibular system has also been implicated in temporal cognition and dynamic attention. The SAMP is that vestibular contributions to speech processing are similar to those it makes in the processing of another sort of structured sound – music. Moreover, just as dance facilitates the perception of rhythm in music, co-speech gestures facilitate the perception of meter in speech. In this section, we describe how precise auditory processing in time relates to the ability to control the body, arguing that vestibular information processing is relevant for movement-related enhancement of dynamic attention. We go on to highlight the sensorimotor theory of rhythm and beat induction (SMT) that suggests metrical accents in music are encoded by the vestibular system as self-motion variables.
4.3.1 Self-Motion Structures Attention
In addition to the classic spatial attention system, cognitive neuroscientists have suggested the brain also utilizes a distinct dynamic attention system to understand rapidly changing stimuli. Dynamic attention relies on the brain’s ability to make predictions about space and time, and serves to facilitate the processing of stimuli that occur both where and when they were predicted (Arnal and Giraud, Reference Arnal and Giraud2012). This is especially true for the processing of sound sequences, in which the brain makes a variety of predictions about their temporal structure.
Mechanistically, the motor system is thought to play an important role in dynamic attention and active perception, and several accounts link temporal aspects of sensory processing with mechanisms that guide the timing of physical actions (Coull and Droit-Volet, Reference Coull and Droit-Volet2018). For example, while temporal regularity improves performance on target detection tasks (e.g., Jones et al., Reference Jones, Moynihan, MacKenzie and Puente2002), additional performance benefits are observed when participants physically move in time with the stimuli relative to passive listening conditions (e.g., Schmidt-Kassow et al., Reference Schmidt-Kassow, Heinemann, Abel and Kaiser2013; Morillon and Baillet, Reference Morillon and Baillet2017).
Although the precise mechanisms that link motor activity to dynamic attention are unclear, there is growing evidence that some of the benefits associated with auditory-motor synchronization derive from the vestibular system. Rather than asking participants to tap in time with a tone sequence, Schmidt-Kassow et al. (Reference Schmidt-Kassow, Wilkinson, Denby and Ferguson2016) combined the auditory oddball paradigm with the administration of galvanic vestibular stimulation (GVS), a technique used to directly stimulate the vestibular afferent nerve (Dlugaiczyk et al., Reference Dlugaiczyk, Gensberger and Straka2019). Administering subliminal GVS pulses either in or out of phase with simple auditory tone sequences, these investigators observed P300 enhancement when vestibular stimulation was temporally aligned with the auditory tones, but not when stimulation was out of phase (Schmidt-Kassow et al., Reference Schmidt-Kassow, Wilkinson, Denby and Ferguson2016). This finding suggests the vestibular system makes an independent contribution to the enhanced processing in this paradigm.
The patient literature also points to a critical role for the vestibular system in dynamic attention (Bigelow and Agrawal, Reference Bigelow and Agrawal2015). Populations with vestibular dysfunction show abnormal P300 responses in oddball detection tasks (e.g., El-Gharib et al., Reference El-Gharib, Nada and Lasheen2018; Wang et al., Reference Wang, Huang and Feng2022). Moreover, studies carried out during spaceflight suggest that endogenous attentional control is gravity-dependent – and thus may recruit vestibular processing resources (e.g., Salatino et al., Reference Salatino, Iacono and Gammeri2021; Takács et al., Reference Takács, Barkaszi and Czigler2021).
4.3.2 Dynamic Attention to Rhythms in Music
Interactions between motor activity and dynamic attention are also relevant for the perception of rhythm in music, in which people lock onto an underlying frequency, referred to as the beat. In fact, some theories treat rhythm perception as mediated by the vestibular system (Todd, Reference Todd1999; Todd and Lee, Reference Todd and Lee2015). According to the SMT of rhythm and beat induction, rhythm perception is a form of sensory-guided action, such that the sensory components initiating body movements are vestibular mechanisms that contribute to the representation of the self in space.
The SMT motivates predictions about human rhythmic capacities that align well with principles of vestibular physiology. The so-called phase transition phenomenon shows how beat perception spontaneously adjusts to the note value (whole, half, quarter note, etc.) that people can most consistently move their body to in time (1–2 Hz) (Haegens and Zion Golumbic, Reference Haegens and Zion Golumbic2018). Notably, this frequency (~1.7 Hz) is a sort of “Goldilocks” rate for the integrity of a variety of otolith-driven reflexes (MacDougall and Moore, Reference MacDougall and Moore2005), and is also the optimal rate for performance on auditory oddball tasks (Zalta et al., Reference Zalta, Petkoski and Morillon2020).
The SMT also suggests that movement-driven enhancement of rhythm perception results because head movement stimulates the vestibular system. Accordingly, research with infants and adults shows it is possible to bias the perception of ambiguous rhythms (i.e., rhythms without physical accents that demarcate beat periods) to align in time with either phasic stimulation of the vestibular nerve or physical movement of the head (Phillips-Silver and Trainor, Reference Phillips-Silver and Trainor2005, Reference Phillips-Silver and Trainor2008; Trainor et al., Reference Trainor, Gao, Lei, Lehtovaara and Harris2009). These studies support a critical role of vestibular information processing for musical rhythm perception.
Beyond this, while acceleration of the head is the primary stimulus for type I receptors in the otoliths, bone-conducted and air-conducted (sound) vibrations can also stimulate the vestibular afferents (see Curthoys et al., Reference Curthoys, Grant and Pastras2019, for a review). Vestibular acoustic sensitivity is restricted to relatively low frequencies (300–1,000 Hz), thus offering a potential explanation for why low-frequency tones are more likely to be interpreted as marking the beat structure of rhythms (Møller et al., Reference Møller, Stupacher, Celma-Miralles and Vuust2021), improve synchronization performance, and elicit spontaneous sensorimotor entrainment (e.g., Varlet et al., Reference Varlet, Williams and Keller2020).
4.3.3 Self-Motion in Musical Rhythms
Predictive coding perspectives treat physical action as the brain’s attempt to minimize errors between predicted and actual incoming sensory inputs, rather than the result of motor commands per se (Adams et al., Reference Adams, Shipp and Friston2013). The SMT suggests that in addition to auditory and proprioceptive predictions, the brain issues vestibular predictions. Because the otolith organs encode information about forces experienced by the head, one way to ascertain the involvement of the vestibular system in rhythm perception is to measure the timing and magnitude of force parameters during dance.
If vestibular codes are recruited for rhythm perception, the body movements that track the beat should be those that signal changes in forces experienced by the head and body. When people hear rhythms, they tend to “bounce” regularly in ways that recruit the head or the trunk (e.g., Toiviainen et al., Reference Toiviainen, Luck and Thompson2010; Janata et al., Reference Janata, Tomic and Haberman2012) and thereby involve vestibular stimulation. Rhythmic arm movements often involve the controlled timing of variations in velocity, that is, transient periods of acceleration and deceleration especially along the vertical axis (Luck and Sloboda, Reference Luck and Sloboda2009; Colley et al., Reference Colley, Varlet, MacRitchie and Keller2018), and these too are likely registered by the vestibular system. In this way, nonlocal consequences of limb movement can be reflected in changes in the force the ground exerts on the support surface of the body – a measurement called the vertical ground reaction force (GRF).
Consistent with a vestibular encoding for rhythm, the spatial properties of arm movements during dance are less relevant for tracking the beat than their force-related properties. For example, when asked to judge whether a dancer’s movements were synchronized to music, participants seemed to exploit the dancer’s vertical GRF (Takehana et al., Reference Takehana, Uehara and Sakaguchi2019). The perception of synchrony was apparently unrelated to purely spatiotemporal descriptions of their movements and was instead related to the way the dancer’s arm movements transferred momentum across the body (Takehana et al., Reference Takehana, Uehara and Sakaguchi2019).
In sum, when people encounter musical rhythms, the brain not only issues temporally organized predictions about impending auditory events but also predictions that are interoceptive in nature and explicitly tied to self-motion. An afferent vestibular prediction shifted forward in time is what “leads” the body to act out rhythms. Likewise, the perception of audiovisual synchrony in dance suggests a multimodal encoding of rhythm that is sensitive to the force of the dancer’s movements (see also Su, Reference Su2014).
4.4 Vestibular Rhythms in Speech
In the previous section, we reviewed evidence illustrating how temporal predictions – especially those facilitated by movement – interact with attentional gain and can enhance the perception of target stimuli presented periodically or in musical rhythms. In this section, we consider how vestibular information processing may also be relevant for structuring attention to speech.
In noting the potential relevance of articulatory gestures of the face and the body for the perception of speech, the SAMP is in some ways reminiscent of the motor theory of speech perception. Through various iterations, the motor theory involves the proposal that speech is perceived by direct apprehension of a speaker’s vocal and articulatory gestures (Liberman et al., Reference Liberman, Cooper, Shankweiler and Studdert-Kennedy1967), and involves the recruitment of the motor system (Galantucci et al., Reference Galantucci, Fowler and Turvey2006). Although speech perception is best construed as subserved by the auditory system, it involves coordination with other neural processing systems (see Holt and Peele, Reference Holt, Peelle, Holt, Peelle, Coffin, Popper and Fay2022, for a review). In accordance with modern accounts of active inference (e.g., Adams et al., Reference Adams, Shipp and Friston2013), the SAMP postulates that any embodied contribution to speech perception would be served in part by dynamic activity in systems relevant for vocalization.
As argued below, the claim in the SAMP is that the vestibular system aids the apprehension of metrical structure, thus influencing the perception of speech. Below, we describe the biomechanical link between the peripheral organs in the vestibular system and the act of speaking. Because speech stimulates the otoliths, it sets up the need for the vestibular system to model the acoustic energy in terms of its impact on self-motion variables.
4.4.1 Vestibular Rhythms in Prosody
Vibrational energy from the larynx during vocalization offers a mechanical stimulus above the sensitivity threshold for the vestibular end organs (Curthoys et al., Reference Curthoys, Grant and Pastras2019), raising the possibility that they play some role in regulating speech. Trivelli et al. (Reference Trivelli, Potena and Frari2013) found that the integrity of saccular function, that is, neural activity in the vestibular system that encodes vertical head movement, predicts speech rehabilitation success in individuals with bilateral sensorineural hearing loss. They suggest that in the absence of normal hearing, the vestibular apparatus may play a direct role in regulating vocal behavior, and that vocal-mechanical imprints sensed by the saccule could serve as a target for speech sound representations and facilitate aspects of articulation (see also Sheykholeslami and Kaga, Reference Sheykholeslami and Kaga2002).
The biomechanical link between the larynx and the vestibular periphery raises the possibility that some aspect of vestibular information is directly encoded in speech, perhaps because speech alters neural precision weighting in the vestibular system (Weissman et al., Reference Weissman, Ekelman, DiScenna and Leigh1989; Diaz-Artiles and Karmali, Reference Diaz-Artiles and Karmali2021). Analogous to the way loud bass tones in musical rhythms would be registered and thus modeled by the vestibular system, it is likely that the vestibular system also models the physical consequences of the act of speaking (Curthoys et al., Reference Curthoys, Grant and Pastras2019), and this may be particularly relevant for the neural representation of prosody.
A speaker’s prosody, that is, modulations of amplitude, duration, or pitch accents in the speech envelope, involves the expression of intonation and lexical stress. The timescale of important prosodic events is well suited for vestibular encoding. Acoustic correlates of prosody are slower than the timescale of phonetic or syllabic information and unfold at a rate (< 3 Hz) suitable for interactions between motor activity and attention, as well as hypothesized comparisons between rhythmic processing in speech and music (Gordon et al., Reference Gordon, Magne and Large2011; Magne et al., Reference Magne, Jordan and Gordon2016; see Chapter 35). In particular, the rate of variation in vocal stress may support “rhythmic attending” to spoken language, akin to the pulse in musical rhythms (see Haegens and Zion Golumbic, Reference Haegens and Zion Golumbic2018).
Because the activation of the vestibular system results from sonorant (voiced) properties of speech, vowels would disproportionately be registered by the vestibular organs. Accordingly, a study using the tapping task showed that participants tap at the onset of vowels, especially metrically strong vowels (Rathcke et al., Reference Rathcke, Lin, Falk and Bella2021; see Chapter 2). Just as participants in rhythm perception studies tap with greater force on strong beats than on weak ones (Benedetto and Baud-Bovy, Reference Benedetto and Baud-Bovy2021), speakers tap more forcefully when they produce stressed syllables than unstressed ones (Parrell et al., Reference Parrell, Goldstein, Lee and Byrd2014).
As the natural alignment between motor behavior and auditory events reveals the dynamics of attentional gain, there are performance benefits for detecting target speech sounds or mispronunciations that occur on stressed syllables relative to unstressed syllables (e.g., Cole and Jakimik, Reference Cole and Jakimik1980). Performance on phoneme detection and syntactic judgment tasks improves if sentences are preceded by a rhythmic prime that matches the prosodic structure of the upcoming speech (Cason et al., Reference Cason, Astésano and Schön2015; see also Chapters 22 and 25). Moreover, the detection of target words improves when participants actively tap in time with stressed syllables relative to unstressed syllables in naturally spoken sentences (Falk and Dalla Bella, Reference Falk and Dalla Bella2016).
In keeping with a vestibular encoding, evidence suggests that mechanisms associated with “rhythmic attending” are sensitive to the vocal energy patterns in speech. Although speech lacks the stable periodicity that defines rhythms found in music (Dalla Bella et al., Reference Dalla Bella, Białuńska and Sowiński2013; see also Chapter 26), it may nonetheless benefit from mechanisms that support precise sensorimotor synchronization and auditory processing in time (Ladányi et al., Reference Ladányi, Persici, Fiveash, Tillmann and Gordon2020), including sensitivity to the body movements that accompany speech.
4.4.2 Prosody across the Body
In natural communication settings, discourse is an exhibition of fully embodied activity. The prominent phase of co-speech-gesture activity is often mirrored in dynamic changes of suprasegmental components of prosody (McNeill, Reference McNeill, Sebeok and Umiker-Sebeok1995; Loehr, Reference Loehr2007). Machine-learning algorithms trained on multimodal discourse corpora can synthesize remarkably natural co-speech movements from novel speech input – bolstering the idea that vocal acoustics reflect information about the speaker’s concurrent bodily motion (Bozkurt et al., Reference Bozkurt, Yemez and Erzin2016). In particular, the apex of manual gesture activity often coincides with salient prosodic contours in the speech signal (e.g., Leonard and Cummins, Reference Leonard and Cummins2011). This correspondence is perhaps most apparent when speakers use beat gestures – downward, pulsing movements of the hand to signal emphasis – and is generally invariant to a gesture’s communicative function (e.g., Rochet-Capellan et al., Reference Rochet-Capellan, Laboissière, Galván and Schwartz2008).
Importantly, co-speech gestures that reflect the rise and fall of the speech signal manifest across the entire body, including movements of the torso and the head (Cvejic et al., Reference Cvejic, Kim and Davis2010; Kim et al., Reference Kim, Cvejic and Davis2014). It has been estimated that the head moves around 90% of the time when people talk (Hadar et al., Reference Hadar, Steiner, Grant and Rose1983), and the apex in motion coincides with peaks in amplitude and/or pitch, such that alignment prefers phrasal, rather than syllabic, components of speech (e.g., Krahmer and Swerts, Reference Krahmer and Swerts2007). The rhythmic body movements accompanying speech may thus be understood as aimed at minimizing prediction error arising from the vestibular impact of speaking, as moments of vocal stress are accompanied by changes in head acceleration.
The natural relationship between co-speech movements and speech is a deeply embedded feature of spoken language and affects speech processing in several ways, such as enhancing speech-in-noise comprehension (Munhall et al., Reference Munhall, Jones, Callan, Kuratate and Vatikiotis-Bateson2004) and facilitating speech segmentation (Kitamura et al., Reference Kitamura, Guellaï and Kim2014). Visual correlates of prosody are even robust enough to shape how speakers are ultimately heard by others. The relative timing of co-speech gestures can bias the perception of spoken emphasis (lexical stress) (e.g., Treffner et al., Reference Treffner, Peter and Kleidon2008), and when lexical stress patterns determine their identity – for example, in the contrast between the English verb “object” (ob-JECT) and the noun “object” (OB-ject) – spoken word recognition is systematically influenced by the exact moment that simple beat gestures land during listening (Bosker and Peeters, Reference Bosker and Peeters2021).
Besides lexical identity, prosodic cues can serve to disambiguate phrase structure during sentence comprehension. In a study that compared the relative influence of acoustic versus gestural timing on the processing of syntactically ambiguous phrases, Guellaï and colleagues found that listeners give more weight to the timing of co-speech gestures than to speech acoustics (Guellaï et al., Reference Guellaï, Langus and Nespor2014). The timing of co-speech gestures thus influences linguistic processing of the speech at multiple different levels and does so in a way that coincides with prosodic cues in the acoustic signal.
Whereas previous scholars have used the phrase “visual prosody” to describe how the kinematics of co-speech movements abstractly mirror intonational or stress fluctuations in the speech signal (Graf et al., Reference Graf, Cosatto, Strom and Huang2002), we prefer multimodal prosody. The use of “visual” presumes that prosody is an acoustic phenomenon supported by gestural information, while the use of “multimodal” is meant to emphasize prosody’s inherent basis in multiple modalities. For the speaker, speech is at least both an acoustic stimulus and a vibrotactile one (see Orepic et al., Reference Orepic, Kannape, Faivre and Blanke2023). The SAMP suggests prosodic rhythms, both seen and heard in face-to-face conversation, recruit a perceptual model grounded in a multimodal representation of the speaking body that also includes vestibulo-motor information.
4.5 A Sensorimotor Account of Multimodal Prosody
The remainder of this chapter will outline the SAMP. The SAMP suggests that during typical face-to-face interactions, the temporal mapping of speaker emphasis may be extracted through both auditory and visual channels together with vestibular codes regarding the inertial motion of the speaker. Cross-modal integration between speech and gesture in communicative contexts might reflect vestibular information processing used for audiovisual binding in the context of full-body activity – especially regarding the way speech is bound to the visual constituents of prosody. Sensitivity to this information may act as a driving force and a functional bridge that promotes attentional alignment between individuals during communication.
4.5.1 Self-Motion in Speaking
One hint that the vestibular system might mediate the production of co-speech gestures comes from patient data that highlight a dissociation between the neurocognitive mechanisms normally important for motor control and those for communicative movements. Consider, for example, the deafferented patient IW who suffers from a neurological condition that resulted in a loss of his proprioceptive sense from the neck down (Cole and Sedgwick, Reference Cole and Sedgwick1992). Although IW relies heavily on vision for normal motor control in space, his co-speech gestures are naturally coordinated in time with his speech – even when his own body is restricted from view (McNeill, Reference McNeill, Sebeok and Umiker-Sebeok1995). As IW has largely intact vestibular reflexes (see Day and Cole, Reference Day and Cole2002, for a discussion), his spared ability to speak and gesture with relative fluidity may be due to information processing involving acoustic-vestibular integration.
A key tenet of the SAMP is that speech production is an action taken by the entire body that requires predictions about its afferent consequences that are at least both acoustic and vestibular in nature. Because the vestibular system models how movements of the body engender change sensed by the semicircular canals and otoliths, the sensory “target” associated with expressive co-speech movements could feasibly be tied to self-motion variables that reflect various biomechanical consequences of vocal expression. We have already seen that one such target involves the physical impact of vocalization on the vestibular system. Another is the way that co-speech gestures affect the acoustic parameters of speech (Pouw and Fuchs, Reference Pouw and Fuchs2022). The correlation in visual and acoustic aspects of the prosodic signals arises at least in part from the fact that strong arm movements perturb the respiratory system and can modulate the amplitude (and occasionally the pitch) of the speech signal (Pouw et al., Reference Pouw, Harrison and Dixon2020).
But in addition to their impact on speech production, arm movements also affect balance. Particularly for downward movements, momentum transfer from upper limb activity mandates muscle activity in the torso to maintain postural control. Just as a kinetic (force-related) description of overt dance movements revealed how musical rhythms were embodied, kinetic properties of co-speech movements are also relevant for their temporal relation to continuous speech (Tuite, Reference Tuite1993). As noted above, speaking often occurs with movement of the head (Tiede et al., Reference Tiede, Mooshammer and Goldstein2019), and the degree of physical impulses through manual gesture correlates with the magnitude of concurrent head movements (Pouw et al., Reference Pouw and Fuchs2022). We suggest that the correlated nature of head and arm movements is not a coincidence, but rather arises because co-speech movements are partially designed to create a mechanical stimulus detectable by the vestibular periphery.
4.5.2 Modeling Self-Motion during Multimodal Discourse Comprehension
From the perspective of comprehension, the SAMP suggests the integration of speech and gestures should benefit from perceptual mechanisms that compute information about forces experienced by the speaking body. Notably, this would include the way that our physical movements are affected by gravity. Evidence suggests the vestibular system contributes to an internal model of gravity critical for mediating perception and action in time (see Jörges and López-Moliner, Reference Jörges and López-Moliner2017, for a review). Because the force-related properties of movement are relevant for respiration and vocal expression, the brain’s modeling of various force parameters might be recruited when co-speech movements are perceptually relevant for the apprehension of speech.
The vestibular system might also contribute to gesture comprehension in a manner inspired by traditional notions of embodied simulation (e.g., Gallese, Reference Gallese2005). In other words, vestibular codes relevant for movement control may be reused to represent self-motion information associated with the observed actions of other people (Jeannerod, Reference Jeannerod1994). Dynamic representations of self-motion information encoded by the vestibular periphery represent not only current, veridical movements of the head or body but also hypothetical ones (Mast and Ellis, Reference Mast and Ellis2015). Dynamic percepts that exhibit self-motion, such as other people, would surely be a sensible target for this type of function (Deroualle and Lopez, Reference Deroualle and Lopez2014; Lopez et al., Reference Lopez, Falconer, Deroualle and Mast2015). Vestibular resources involved in the dynamic modeling of self-motion information might also be exploited for perception of co-speech gestures, especially so-called beat gestures that coincide with the rhythmic qualities of speech. Particularly in rich, communicative contexts, constructs amenable to interpretation by the vestibular system may be precisely what the brain is motivated to decode from audio and visual signals.
4.5.3 Multisensory Integration and Dynamic Attention
This review has highlighted work across several disciplines showing how the vestibular system is relevant for various aspects of temporal cognition and dynamic attentional control. As such, it could serve as a critical axis for embodied engagement with sensory information – including the quasi-rhythmic information in speech. For example, motor-related oscillatory dynamics are implicated in structuring temporal predictions in the context of musical rhythms, speech, and speech-gesture integration during discourse processing (Arnal et al., Reference Arnal, Doelling and Poeppel2015; Keitel et al., Reference Keitel, Gross and Kayser2018; Biau et al., Reference Biau, Schultz, Gunter and Kotz2022; see also Chapter 13). In these contexts, the vestibular system likely contributes to motor-driven temporal predictions that improve how attention is structured in time.
Vestibular information processing is especially relevant in multimodal interactional contexts between embodied agents. Above, we noted cross-modal influences on the perception of talkers. Seeing a speaker’s gestures can influence the perception of metrical aspects of their speech. The SAMP suggests this is because body movements engage a common underlying action representation relevant for sensorimotor timing. Moreover, perceptual inference about inertial properties of biological motion, such as the vertical GRF, may direct cross-modal integration in ways that are relevant for the perception of meter in speech (see Figure 4.2).
Force transfer and multimodal prosody.
Graphical depiction of audio and visual components of naturalistic speech recorded using a motion capture setup. Top: the vertical acceleration for four different mocap markers. Below: estimate of the speaker’s vertical GRF. The lowpass envelope is superimposed. Bottom: raw speech signal.
The figures above show the speaker at each time frame associated with the time points demarcated by the vertical lines across the four plots (see x-axis). The stressed syllable in “ground” manifests closely in time with the contour of the GRF estimate – a function of the peak in downward acceleration of the speaker’s head, trunk, and arm movements.
Figure 4.2 Long description
A dual-axis multiline graph compares low-pass envelope and vertical acceleration. It plots a combination of five solid and dashed lines for Env, head, torso, right wrist, left wrist, and vertical G R F. These lines fluctuate horizontally. Below it, another dual-axis multiline graph compares the lowpass envelope and the estimated vertical ground reaction force. It plots a solid and a dashed line for Env and torso. These lines fluctuate horizontally. At the bottom, a raw waveform is depicted. The horizontal axis represents time and the vertical axis represents amplitude. The waveform displays a complex, fluctuating signal over a period of approximately 1.25 seconds. The legends for Env, head, torso, right wrist, left wrist, and vertical G R F are given at the bottom right.
4.6 Conclusions
This chapter suggests that neural mechanisms relevant for vestibular information processing are likely recruited for the neural representation of speech, especially its prosodic qualities. Vestibular-sensitive mechanisms may assist in coding latent variables about speaker movement during the processing of co-speech gestures, during the processing of unaccompanied speech, and during the processing of biological motion. The SAMP is that the multimodal integration in the prosodic signal arises via the convergence of visual information conveyed by head and full-body movements with acoustic information conveyed in the prosodic contours of speech via a common vestibulo-motor representation of the speaker’s behavior.
Although we have focused on the theoretical implications of the SAMP for the mechanisms involved in structuring attention to multimodal discourse, the proposal also has clinical implications. Given the high comorbidity rates between problems with speech-language, motor-timing, and rhythmic faculties (e.g., Fiveash et al., Reference Fiveash, Bedoin, Gordon and Tillmann2021), clinical researchers should explore vestibular contributions to language functions. In particular, speech scientists should investigate the integrity of peripheral or central vestibular mechanisms in these populations. Such work will allow us to develop a more detailed understanding of vestibular contributions to the production and comprehension of speech and may lead to the discovery of clinical interventions. In the meantime, theoretical details are not necessary for the identification of reliable biomarkers useful for early diagnosis and therapeutic routing (see Efimova and Nikolaeva, Reference Efimova and Nikolaeva2020).
Summary
We present a novel hypothesis regarding the neural mechanisms important for speech-gesture integration and attention to discourse. The SAMP suggests that neural codes for the inertial motion of the body are recruited for the representation of speech, and support the encoding and decoding of the prosodic features of multimodal discourse.
Implications
Dynamic changes of suprasegmental components of spoken prosody are often mirrored in prominent aspects of the speaker’s co-speech gestures. Considering speech as an action taken by the entire body offers new ways to understand how rhythms are embodied in linguistic and musical contexts.
Gains
Considering the biophysical connection between the vestibular system and speech together with evidence for vestibular contributions to cognition, the SAMP motivates novel lines of research regarding the neurobiology of spoken language. We highlight the potential to identify biomarkers for atypical language and rhythm development.
Open access